@lossless.org/objectstorage 9.0.0

package/readme.md

@@ -0,0 +1,1430 @@
+# @lossless.org/objectstorage
+
+A high-performance, S3-compatible storage server powered by a **Rust core** with a clean TypeScript API. Runs standalone for dev/test — or scales out as a **distributed, erasure-coded cluster** with QUIC-based inter-node communication. No cloud, no Docker. Just install the package and go. 🚀
+
+## Issue Reporting and Security
+
+For reporting bugs, issues, or security vulnerabilities, please visit [community.foss.global/](https://community.foss.global/). This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a [code.foss.global/](https://code.foss.global/) account to submit Pull Requests directly.
+
+## Moving from @push.rocks/smartstorage
+
+Install `@lossless.org/objectstorage` and update imports and the public API names:
+
+| Previous name | New name |
+| --- | --- |
+| `@push.rocks/smartstorage` | `@lossless.org/objectstorage` |
+| `SmartStorage` | `ObjectStorage` |
+| `ISmartStorage*` / `TSmartStorage*` | `IObjectStorage*` / `TObjectStorage*` |
+| `SmartStorageResourceFenceError` | `ObjectStorageResourceFenceError` |
+| `computeSmartStorageResourceFenceEffectivePayloadSha256V1` | `computeObjectStorageResourceFenceEffectivePayloadSha256V1` |
+| `smartStorageResourceFenceErrorCodes` | `objectStorageResourceFenceErrorCodes` |
+| Prometheus `smartstorage_*` metric families | Prometheus `objectstorage_*` metric families |
+| Test-only `SMARTSTORAGE_RESOURCE_FENCE_TEST_*` variables | `OBJECTSTORAGE_RESOURCE_FENCE_TEST_*` variables |
+
+Update monitoring queries and dashboards alongside the package. The new module
+exports the new API names without aliases for the old names. Generic types such
+as `IStorageConfig`, `IBucketExport`, and `IClusterHealth` keep their names.
+
+The rename preserves the current storage and cluster protocols. Existing bucket
+roots, credentials, retention state, migration receipts, and resource-fencing
+identities keep their byte formats. Versioned `smartstorage.*` format identifiers
+and hash domains, `.smartstorage*` filesystem names, retention metadata keys,
+the internal streaming-verification header, and the QUIC `smartstorage` protocol
+identity remain unchanged. Do not rewrite these values in existing data.
+The native `ruststorage` executable and its Linux amd64/arm64 artifact names also
+remain unchanged. Earlier storage-layout migrations retain their existing rules
+described below.
+
+## Why objectstorage?
+
+| Feature | objectstorage | MinIO | s3rver |
+|---------|-------------|-------|--------|
+| Install | `pnpm add` | Docker / binary | `npm install` |
+| Startup time | ~20ms | seconds | ~200ms |
+| Large file uploads | Streaming, bounded memory | Yes | OOM risk |
+| Range requests | Seek-based | Yes | Full read |
+| Language | Rust + TypeScript | Go | JavaScript |
+| Multipart uploads | ✅ Full support | Yes | No |
+| Auth | AWS SigV4 (full verification) | Full IAM | Basic |
+| Bucket policies | IAM-style evaluation | Yes | No |
+| Clustering | ✅ Erasure-coded, QUIC | Yes | No |
+| Multi-drive awareness | ✅ Per-drive health | Yes | No |
+
+### Core Features
+
+- 🦀 **Rust-powered HTTP server** — hyper 1.x with streaming I/O, bounded buffering, backpressure
+- 📦 **S3-compatible API** — supported operations work with AWS SDK v3 and SmartBucket
+- 💾 **Filesystem-backed storage** — buckets map to directories, objects to files
+- 📤 **Streaming multipart uploads** — large files with bounded memory use
+- 📐 **Byte-range requests** — `seek()` directly to the requested byte offset
+- 🔐 **AWS SigV4 authentication** — full signature verification with constant-time comparison
+- 📋 **Bucket policies** — IAM-style JSON policies with Allow/Deny evaluation and wildcard matching
+- 🌐 **CORS middleware** — configurable cross-origin support
+- 🧹 **Clean slate mode** — wipe storage on startup for test isolation
+- 📊 **Runtime storage stats** — cheap bucket summaries and global counts without S3 list scans
+- 🔑 **Runtime credential rotation** — list and replace active auth credentials without mutating internals
+- 🧩 **Bucket tenants** — provision one scoped S3 credential per bucket with restart persistence
+- ⚡ **Test-first design** — start/stop in milliseconds, no port conflicts
+
+### Clustering Features
+
+- 🔗 **Erasure coding** — Reed-Solomon (configurable k data + m parity shards) for storage efficiency and fault tolerance
+- 🚄 **QUIC transport** — multiplexed, encrypted inter-node communication via `quinn` with zero head-of-line blocking
+- 💽 **Multi-drive awareness** — each node manages multiple independent storage paths with health monitoring
+- 🩺 **Cluster health introspection** — query native node, drive, quorum, and healing status for product dashboards
+- 🤝 **Cluster membership** — static seed config + runtime join, heartbeat-based failure detection
+- ✍️ **Quorum writes** — data is only acknowledged after k+1 shards are persisted
+- 📖 **Quorum reads** — reconstruct from any k available shards, local-first fast path
+- 🩹 **Self-healing** — background scanner detects and reconstructs missing/corrupt shards
+
+## Installation
+
+```bash
+pnpm add @lossless.org/objectstorage -D
+```
+
+> **Note:** The package ships with precompiled Rust binaries for `linux_amd64` and `linux_arm64`. No Rust toolchain needed on your machine.
+
+## Quick Start
+
+### Standalone Mode (Dev & Test)
+
+```typescript
+import { ObjectStorage } from '@lossless.org/objectstorage';
+
+// Start a local S3-compatible storage server
+const storage = await ObjectStorage.createAndStart({
+  server: { port: 3000 },
+  storage: { cleanSlate: true },
+});
+
+// Create a bucket
+await storage.createBucket('my-bucket');
+
+// Get connection details for a supported S3 client
+const descriptor = await storage.getStorageDescriptor();
+// → { endpoint: 'localhost', port: 3000, accessKey: 'STORAGE', accessSecret: 'STORAGE', useSsl: false }
+
+// When done
+await storage.stop();
+```
+
+### Cluster Mode (Distributed)
+
+Cluster mode requires the control root and every drive root to exist before
+startup. The process user must own them, no path component may be a symlink,
+and group/world write permission must be disabled.
+
+```typescript
+import { ObjectStorage } from '@lossless.org/objectstorage';
+
+const storage = await ObjectStorage.createAndStart({
+  server: { port: 3000 },
+  storage: {
+    directory: '/var/lib/objectstorage/control',
+    cleanSlate: false,
+  },
+  cluster: {
+    enabled: true,
+    nodeId: 'node-1',
+    quicPort: 4000,
+    seedNodes: ['192.168.1.11:4000', '192.168.1.12:4000'],
+    erasure: {
+      dataShards: 4,      // k: minimum shards to reconstruct data
+      parityShards: 2,    // m: fault tolerance (can lose up to m shards)
+    },
+    drives: {
+      paths: ['/mnt/disk1', '/mnt/disk2', '/mnt/disk3'],
+    },
+  },
+});
+```
+
+Objects are automatically split into chunks (default 4 MB), erasure-coded into 6 shards (4 data + 2 parity), and distributed across drives/nodes. Any 4 of 6 shards can reconstruct the original data.
+
+## Configuration
+
+All config fields are optional — sensible defaults are applied automatically.
+
+```typescript
+import { ObjectStorage, IObjectStorageConfig } from '@lossless.org/objectstorage';
+
+const config: IObjectStorageConfig = {
+  server: {
+    port: 3000,              // Default: 3000
+    address: '0.0.0.0',      // Default: '0.0.0.0'
+    silent: false,           // Default: false
+    region: 'us-east-1',     // Default: 'us-east-1' — used for SigV4 signing
+  },
+  storage: {
+    directory: './my-data',  // Default: .nogit/bucketsDir
+    cleanSlate: false,       // Default: false — set true to wipe on start
+    pool: {                  // Optional explicit identity for this process' one pool
+      id: 'fast-local',
+      directory: './my-data',
+      backend: { kind: 'localFs' },
+    },
+  },
+  auth: {
+    enabled: false,          // Default: false
+    credentials: [{
+      accessKeyId: 'MY_KEY',
+      secretAccessKey: 'MY_SECRET',
+    }],
+  },
+  cors: {
+    enabled: false,          // Default: false
+    allowedOrigins: ['*'],
+    allowedMethods: ['GET', 'POST', 'PUT', 'DELETE', 'HEAD', 'OPTIONS'],
+    allowedHeaders: ['*'],
+    exposedHeaders: ['ETag', 'x-amz-request-id', 'x-amz-version-id'],
+    maxAge: 86400,
+    allowCredentials: false,
+  },
+  logging: {
+    level: 'info',           // 'error' | 'warn' | 'info' | 'debug'
+    format: 'text',          // 'text' | 'json'
+    enabled: true,
+  },
+  limits: {
+    maxObjectSize: 5 * 1024 * 1024 * 1024, // 5 GB
+    maxMetadataSize: 2048,
+    requestTimeout: 300000,  // 5 minutes
+  },
+  multipart: {
+    expirationDays: 7,             // Must be greater than zero
+    cleanupIntervalMinutes: 60,    // Must be greater than zero
+  },
+  cluster: {                 // Optional — omit for standalone mode
+    enabled: true,
+    nodeId: 'node-1',        // Auto-generated UUID if omitted
+    quicPort: 4000,          // Default: 4000
+    seedNodes: [],           // Addresses of existing cluster members
+    erasure: {
+      dataShards: 4,         // Default: 4
+      parityShards: 2,       // Default: 2
+      chunkSizeBytes: 4194304, // Default: 4 MB
+    },
+    drives: {
+      paths: ['/mnt/disk1', '/mnt/disk2'],
+    },
+    heartbeatIntervalMs: 5000,  // Default: 5000
+    heartbeatTimeoutMs: 30000,  // Default: 30000
+  },
+};
+
+const storage = await ObjectStorage.createAndStart(config);
+```
+
+### Storage Pools and Mounted-Filesystem Serving
+
+One ObjectStorage process serves one storage pool. Existing `storage.directory`
+configuration remains compatible and normalizes to a `default` `localFs` pool.
+For `localFs`, `storage.directory` and `storage.pool.directory` must resolve to
+the same path when both are supplied. For `mountedFs`, both values must instead
+be lexically exact absolute mountpoints and identical strings. A root is durably
+bound to its pool ID and cannot later be reassigned to a different pool.
+
+For standalone `localFs` pools, ObjectStorage creates a missing root with `0700`
+permissions. An existing root must be owned by the process user and must not be
+group- or world-writable; startup fails before opening the S3 listener when this
+contract is not met. ObjectStorage never silently changes an existing root's
+ownership or permissions.
+
+On Linux, standalone ObjectStorage can serve an exact NFS or SMB mount. The
+configured directory itself must be the absolute mountpoint (not a symlink or
+canonical alias), `expectedSource` must contain a numeric IPv4 or IPv6 address,
+the client mount must enable the Linux `nosymfollow` VFS option, and
+`cleanSlate` is forbidden. Mounted pools are not supported in cluster mode.
+Onebox and other mount-provisioning clients must include `nosymfollow` in their
+NFS or CIFS mount options before ObjectStorage starts. NFS mounts must keep
+server-coordinated locking enabled: `nolock` and
+`local_lock=all|flock` are rejected, while an absent `local_lock` option,
+`local_lock=none`, or `local_lock=posix` is accepted because ObjectStorage uses
+`flock(2)` for its provider locks. CIFS mounts must not use `nobrl`.
+Accepted source forms are `IP:/absolute/export` for NFS and
+`//IP/share[/path]` for SMB. IPv6 addresses use brackets, for example
+`[2001:db8::10]:/archive` and `//[2001:db8::10]/archive`.
+
+Startup opens the mountpoint once and confines every standalone storage,
+credential, policy, retention, fencing, and background operation beneath that
+descriptor through `/proc/self/fd`. It verifies the descriptor's mount ID,
+device, NFS/CIFS filesystem magic, and authoritative `ST_NOSYMFOLLOW` VFS flag,
+plus the effective mount and superblock locking options, then rereads the
+complete mount activation before serving. Every storage
+boundary revalidates the activation and `ST_NOSYMFOLLOW`. Any drift permanently
+poisons the process, fails storage-derived HTTP and management boundaries
+closed, never follows descendant symlinks, and never falls back to a host
+directory at the configured path. Lifecycle, liveness, cluster-status, and
+cached-diagnostic operations remain available for shutdown and diagnosis.
+
+Every server mode stops accepting HTTP connections, gracefully drains each
+accepted connection, and joins independently owned request and multipart
+workers before releasing the mount descriptor or root-maintenance lock. The
+drain is bounded to 30 seconds. A timeout is reported as an incomplete stop:
+the Rust process retains the server, descriptors, locks, and unfinished task
+handles so the caller can retry `stop()` after hard-mounted I/O recovers.
+Streamed object reads move their file descriptor into the same owned worker
+lifecycle, so a disconnected client cannot detach a blocked filesystem read.
+Active HTTP request lifecycles are capped at 4,096; excess requests receive a
+retryable `503 Service Unavailable` response with `Retry-After: 1`. A terminal
+shutdown result is reported only after all owned work is complete and remains
+available to retrying stop waiters until the management process terminates.
+
+```typescript
+const storage = await ObjectStorage.createAndStart({
+  storage: {
+    directory: '/mnt/archive',
+    pool: {
+      id: 'archive',
+      directory: '/mnt/archive',
+      backend: {
+        kind: 'mountedFs',
+        expectedFilesystemType: 'nfs',
+        expectedSource: '192.0.2.10:/archive',
+        // IPv6 is canonicalized as: [2001:db8::10]:/archive
+      },
+    },
+  },
+});
+```
+
+Serving startup also runs the root-ownership, same-filesystem,
+server-coordinated lock-contention, directory-fsync, and durable
+provider-identity checks through the descriptor anchor. A mounted runtime
+observation is `productionEligible` after these runtime checks pass and while
+the activation is unpoisoned. Atomic directory exchange is reported separately
+as an optional capability: NFS filesystems without it can still serve S3,
+atomic create-only uploads, fenced bucket ensure/delete, and migration cleanup,
+while atomic whole-bucket replacement and isolated restore remain unavailable
+and fail closed. Its `remoteLocking` check and
+`filesystem.remoteLockingMode` observation distinguish server-coordinated
+locking from unsafe local or disabled locking. `getStoragePoolDiagnostic()`
+remains available after poison and returns cached activation identity and
+poison evidence without accessing the storage filesystem.
+
+Probe an existing exact mount without starting the S3 listener:
+
+```typescript
+const observation = await ObjectStorage.probeStoragePool({
+  id: 'archive',
+  directory: '/mnt/archive',
+  backend: {
+    kind: 'mountedFs',
+    expectedFilesystemType: 'nfs',
+    expectedSource: '192.0.2.10:/archive',
+  },
+});
+
+console.log(observation.supported);
+console.log(observation.productionEligible); // false: this read-only probe skips write semantics
+console.log(observation.checks);
+```
+
+The read-only probe verifies the exact active mount, numeric source,
+descriptor-bound mount ID, device, filesystem magic, `ST_NOSYMFOLLOW`, and
+server-coordinated locking options, and activation stability.
+It reports observed block and transfer sizes but does not change mount tuning or
+write to the target. Mutating semantic checks remain `notRun`, so a successful
+diagnostic probe keeps `supported` at `null` and `productionEligible` at
+`false`; production eligibility is established only by serving startup.
+
+### Common Configurations
+
+**CI/CD testing** — silent, clean, fast:
+```typescript
+const storage = await ObjectStorage.createAndStart({
+  server: { port: 9999, silent: true },
+  storage: { cleanSlate: true },
+});
+```
+
+**Auth enabled:**
+```typescript
+const storage = await ObjectStorage.createAndStart({
+  auth: {
+    enabled: true,
+    credentials: [{ accessKeyId: 'test', secretAccessKey: 'test123' }],
+  },
+});
+```
+
+**CORS for local web dev:**
+```typescript
+const storage = await ObjectStorage.createAndStart({
+  cors: {
+    enabled: true,
+    allowedOrigins: ['http://localhost:5173'],
+    allowCredentials: true,
+  },
+});
+```
+
+## Runtime Credentials
+
+```typescript
+const credentials = await storage.listCredentials();
+
+await storage.replaceCredentials([
+  {
+    accessKeyId: 'ADMINA',
+    secretAccessKey: 'super-secret-a',
+  },
+  {
+    accessKeyId: 'ADMINB',
+    secretAccessKey: 'super-secret-b',
+  },
+]);
+```
+
+```typescript
+interface IStorageCredential {
+  accessKeyId: string;
+  secretAccessKey: string;
+  bucketName?: string;
+  region?: string;
+}
+```
+
+- `listCredentials()` returns the Rust core's current runtime credential set.
+- `replaceCredentials()` swaps the full set atomically and persists it under the storage root. On success, new requests use the new set immediately and the old credentials stop authenticating immediately.
+- Requests that were already authenticated before the replacement keep running; auth is evaluated when each request starts.
+- No restart is required, and runtime-created credentials survive restart unless `storage.cleanSlate` clears the bounded storage contents; the owned root, maintenance-lock inode, and any durable provider/pool identity are preserved.
+- Replacement input must contain at least one credential, each `accessKeyId` and `secretAccessKey` must be non-empty, and `accessKeyId` values must be unique.
+
+## Bucket Tenants
+
+Bucket tenants are designed for platform services that need one bucket and one scoped S3 credential per app. Tenant credentials are enforced by the auth layer before the normal bucket-policy/default-auth pipeline, so a scoped credential cannot list all buckets or access another bucket even when it has a valid SigV4 signature.
+
+```typescript
+const tenant = await storage.createBucketTenant({
+  bucketName: 'workapp-123',
+});
+
+// Directly usable by AWS SDK v3 or env injection
+const client = new S3Client({
+  endpoint: `http://${tenant.endpoint}:${tenant.port}`,
+  region: tenant.region,
+  credentials: {
+    accessKeyId: tenant.accessKeyId,
+    secretAccessKey: tenant.secretAccessKey,
+  },
+  forcePathStyle: true,
+});
+
+console.log(tenant.env.S3_BUCKET);
+console.log(tenant.env.AWS_ACCESS_KEY_ID);
+```
+
+```typescript
+await storage.rotateBucketTenantCredentials({ bucketName: 'workapp-123' });
+await storage.deleteBucketTenant({ bucketName: 'workapp-123', accessKeyId: tenant.accessKeyId });
+const descriptor = await storage.getBucketTenantDescriptor({ bucketName: 'workapp-123' });
+const tenants = await storage.listBucketTenants();
+```
+
+- `createBucketTenant()` creates the bucket if needed and stores a scoped credential for that bucket.
+- `rotateBucketTenantCredentials()` replaces the active scoped credential for the bucket and persists the new credential.
+- `deleteBucketTenant({ bucketName, accessKeyId })` revokes one scoped credential and keeps the bucket.
+- `deleteBucketTenant({ bucketName })` revokes scoped credentials for an existing tenant bucket and deletes that bucket's contents recursively.
+- Tenant credentials can list, read, write, and delete objects in their assigned bucket, but cannot list all buckets, access other buckets, copy from other buckets, delete buckets, or mutate bucket policies.
+- Bucket tenant APIs require `auth.enabled: true`.
+
+## Bucket Backup/Restore
+
+```typescript
+const appBackup = await storage.exportBucket({ bucketName: 'workapp-123' });
+await storage.importBucket({ bucketName: 'workapp-123-restore', source: appBackup });
+```
+
+- `exportBucket()` returns a self-contained `smartstorage.bucket.v1` JSON export with only the selected bucket's objects and object metadata. The legacy whole-bucket management export is intentionally bounded to 10,000 objects, 48 MiB of payload, and 8 MiB of serialized keys plus metadata; use the migration transfer surface for larger buckets.
+- `importBucket()` validates object payload size and MD5 before creating the target bucket if needed, then restores the exported objects into that bucket.
+- Exports do not include credentials, policies, or unrelated tenant data.
+
+## Bucket Migration Control
+
+Standalone storage exposes durable, provider-bound migration control records.
+Discover the local provider identity, receipt key, and cleanup support with
+`getBucketMigrationCapability()`. Create each side with
+`createSourceMigration()` and `createDestinationMigration()`, then advance only
+the explicit source (`active` → `sealed` → `cutoverCommitted` → `retired`) and
+destination (`holding` → `verified` → `cutoverAuthorized` → `published`) paths.
+Before cutover, a destination `aborted` terminal receipt can release a sealed
+source. Every call binds a positive fence token, exact migration binding, and
+opaque controller capability; exact same-token retries replay the durable
+receipt while stale or mismatched requests fail.
+
+Source creation additionally requires `sourceCleanupAccessKeyId`. ObjectStorage
+verifies that it is the bucket's one exact scoped credential, durably binds its
+hash before creating migration state, and revalidates that ownership before
+every source phase transition and before persisting a cleanup request. A
+provider can have only one nonterminal migration per role and logical bucket;
+terminal records remain as history and do not prevent a later migration ID.
+The standalone registry has a finite capacity of 100,000 migration states.
+Terminal receipts and completed-cleanup tombstones continue to consume that
+capacity because they are retained permanently; controllers must treat
+migration IDs as lifecycle records and capacity-plan before exhaustion.
+ObjectStorage serializes registry creation globally and atomically rejects a new
+migration ID when 100,000 states already exist. Exact replays of existing IDs
+remain available at the bound; the provider never deletes history or permits
+manual registry-file reuse to make capacity.
+Registry enumeration budgets durable locks, canonical crash-left atomic
+temporary files, and unrecognized junk separately. This leaves bounded recovery
+headroom at full state capacity without deleting a temporary file another
+process may still own; every partition remains finite and fails closed when its
+own limit is exceeded.
+
+Source sealing drains bucket mutations and blocks later object, multipart,
+policy, bucket, tenant, and fenced mutation paths. Reads, heads, listings,
+policy reads, and multipart listings remain available while sealed, but they
+hold the bucket data-plane read guard so a later retirement cleanup drains every
+accepted source read before deletion. A retained bucket cannot become a
+migration source.
+
+After the destination is durably `published` and the source is durably
+`retired`, the controller can permanently retire the source data:
+
+```typescript
+const sourceState = await source.inspectBucketMigration({ migrationId });
+const destinationState = await destination.inspectBucketMigration({ migrationId });
+
+const cleanup = await source.cleanupSourceMigration({
+  binding,
+  cleanupFenceToken: 1,
+  opaqueCapability: sourceCapability,
+  sourceRetiredReceiptSha256: sourceState!.terminalReceipt!.receiptSha256,
+  destinationPublishedReceipt: destinationState!.terminalReceipt!,
+  resourceFence: {
+    version: 1,
+    scopeId: `${binding.authorityId}.${binding.logicalBucket}`,
+    token: 42,
+    mutationId: `${migrationId}.source-cleanup`,
+    payloadSha256: cleanupIntentSha256,
+  },
+  accessKeyId: sourceTenant.accessKeyId,
+});
+```
+
+`cleanupSourceMigration()` first persists a permanent requested cleanup
+subrecord that binds the exact retired-source receipt, fully verified
+destination-published receipt, cleanup token, resource fence, and hashed access
+key. It then reuses exact fenced deletion to remove bucket data, every matching
+multipart upload, the exact credential, and the bucket policy before persisting
+the canonical cleanup receipt and generic delete-result digest. Multipart
+deletion immediately reconciles the runtime admission counter and cleanup queue.
+The lock order drains reads and mutations before credential revocation.
+
+The controller owns recovery: ObjectStorage never starts source deletion
+automatically. Retry the exact `cleanupSourceMigration()` request after any
+timeout, process exit, or ambiguous response. A requested-only retry repeats
+the exact resource deletion; if deletion was already durably complete, its
+generic receipt is replayed and the migration cleanup record is finalized.
+After completion the request cannot be rebound, the bucket can never be
+recreated or credentialed, and unrelated global credential replacement resumes.
+Generic bucket deletion and multipart abort remain blocked throughout the
+migration seal.
+
+`inspectBucketMigration()` returns the durable phase, terminal receipts, and
+the optional `sourceCleanup` requested/completed subrecord. Terminal receipts
+and completed cleanup receipts remain available across restart.
+`storage.cleanSlate: true` cannot erase or bypass that history: standalone
+startup is refused whenever any durable migration state exists, including a
+terminal receipt or completed-cleanup tombstone.
+
+## Durable Resource Fencing
+
+Standalone storage with `cleanSlate: false` supports monotonic, crash-safe bucket
+replacement and deletion. The legacy immediate mode (`holdPublication` omitted
+or `false`) still requires callers to drain normal writes before issuing a
+fenced mutation.
+
+```typescript
+const fence = {
+  version: 1 as const,
+  scopeId: 'corestore-node-1.workapp-123',
+  token: 42,
+  mutationId: 'delete-workapp-123',
+  payloadSha256: 'a'.repeat(64),
+};
+
+const result = await storage.deleteBucket({
+  bucketName: 'workapp-123',
+  accessKeyId: tenant.accessKeyId,
+  fence,
+});
+```
+
+For control planes that cannot atomically publish the provider result, request a
+durable write-publication hold:
+
+```typescript
+const held = await storage.importBucket({
+  bucketName: 'workapp-123',
+  source: appBackup,
+  accessKeyId: tenant.accessKeyId,
+  fence,
+  holdPublication: true,
+});
+
+// Persist the provider result and update the control-plane resource first.
+const release = await storage.commitResourcePublication({
+  barrier: held.resourcePublicationBarrier!,
+  resourceFenceReceipt: held.resourceFence!,
+});
+```
+
+`holdPublication: true` is part of the effective mutation identity. ObjectStorage
+drains active target-bucket mutations before the destructive provider operation,
+then keeps all new target mutations behind a durable barrier after the provider
+effects and receipt are fully persisted. Mutations include object, multipart,
+bucket, policy, tenant, and credential changes. They receive retryable S3
+`WritePublicationHeld` (`503`, `Retry-After: 1`) or
+`EFENCE_PUBLICATION_HELD` through the management API. Unrelated buckets remain
+available. Reads are not blocked, so this protocol controls write publication;
+it does not hide newly restored data from readers.
+
+The exact mutation replay returns the same barrier. Different and higher-token
+mutations cannot clear it. `commitResourcePublication()` validates the complete
+barrier, the canonical fence receipt, and its 256-bit opaque commit capability;
+it is safe to retry and returns the same release receipt. Treat
+`commitCapability` as a secret: do not log it or persist it outside protected
+control-plane state. A successful commit removes the raw capability from provider
+state and retains only its hash.
+
+There is no timeout, shutdown release, or administrative bypass. A held barrier
+survives restart and is restored before the listener accepts requests. Invalid,
+unknown, oversized, or insecure publication state fails startup closed.
+`replaceCredentials()` is also rejected while any publication remains held.
+Capability discovery keeps `requiresDrain: true` for legacy immediate mode and
+reports `publicationHoldSupported: true`, `publicationHoldVersion: 1`, and
+`publicationHoldRequiresExternalDrain: false` when the provider can enforce the
+hold internally.
+
+When supplied, the exact `accessKeyId` is bound into the durable mutation
+identity. Its delete receipt is committed only after bucket data is durably
+absent, that exact bucket-scoped credential is durably revoked, and the bucket
+policy is durably absent. Retrying the same fence returns the stored receipt.
+Recovery after a process crash repeats the credential and policy cleanup
+idempotently, so `deletedCredentials` can be `0`; a higher fence against an
+already absent bucket can likewise return `bucketDeleted: false`. Completed
+exact deletes return `credentialsPreserved: false` and
+`policyPreserved: false`.
+
+For backward compatibility, omitting `accessKeyId` retains the original fenced
+data-only delete identity and receipt: tenant credentials and bucket policy are
+preserved, and both preservation fields are `true`.
+
+After an exact delete, a new higher-token authority can recreate the same target
+without removing its durable fence history. Call `ensureBucketTenant()` with the
+new fence first, then issue an exact fenced `importBucket()` with the same
+resource token. Ensure receipts use the
+`smartstorage.bucket.ensure.v1` profile and attest the applied access key,
+provider-computed secret hash, and effective region; the secret itself is never
+written to the receipt or fencing registry.
+
+`computeObjectStorageResourceFenceEffectivePayloadSha256V1()` computes the same
+canonical effective payload digest as the Rust provider for replacement,
+exact-delete, and ensure profiles. A completed fenced ensure returns its receipt
+on `descriptor.resourceFence`.
+
+Absent-bucket fenced ensure removes any residual policy before creating the new
+resource; existing-bucket ensure preserves policy and object data. Once a bucket
+has durable fence history, legacy `createBucket()`, `createBucketTenant()`,
+`rotateBucketTenantCredentials()`, `deleteBucketTenant()`, and unfenced
+`importBucket()` mutations fail with `EFENCE_REQUIRED`. Use fenced ensure,
+exact import, and exact delete for its lifecycle.
+
+`replaceCredentials()` is a trusted process-local administrative override, not
+a fenced lifecycle API. If it changes exact-owned credential material, completed
+receipt replay detects the drift and fails closed with an ownership or durable
+state error.
+
+## Health and Metrics APIs
+
+```typescript
+const health = await storage.getHealth();
+const metrics = await storage.getMetrics();
+```
+
+- `getHealth()` reports running state, storage directory and pool observation, auth enabled state, credential counts, bucket count, object count, total bytes, and cluster health.
+- `getMetrics()` returns numeric counters and a Prometheus text snippet for bucket, object, byte, tenant credential, and cluster-enabled metrics.
+
+## Runtime Stats
+
+```typescript
+const stats = await storage.getStorageStats();
+const bucketSummaries = await storage.listBucketSummaries();
+
+console.log(stats.bucketCount);
+console.log(stats.totalObjectCount);
+console.log(stats.totalStorageBytes);
+console.log(bucketSummaries[0]?.name, bucketSummaries[0]?.objectCount);
+```
+
+```typescript
+interface IBucketSummary {
+  name: string;
+  objectCount: number;
+  totalSizeBytes: number;
+  creationDate?: number;
+}
+
+interface IStorageLocationSummary {
+  path: string;
+  totalBytes?: number;
+  availableBytes?: number;
+  usedBytes?: number;
+  pool?: IStoragePoolObservation;
+}
+
+interface IStorageStats {
+  bucketCount: number;
+  totalObjectCount: number;
+  totalStorageBytes: number;
+  buckets: IBucketSummary[];
+  storageDirectory: string;
+  storageLocations?: IStorageLocationSummary[];
+}
+```
+
+- `bucketCount`, `totalObjectCount`, `totalStorageBytes`, and per-bucket totals are logical object stats maintained by the Rust runtime. They count object payload bytes, not sidecar files or erasure-coded shard overhead.
+- objectstorage initializes these values from native on-disk state at startup, then keeps them in memory and updates them when bucket/object mutations succeed. Stats reads do not issue S3 `ListObjects` or rescan every object.
+- Values are exact for mutations performed through objectstorage after startup. Direct filesystem edits outside objectstorage are not watched; restart the server to resync.
+- `storageLocations` is a cheap filesystem-capacity snapshot. Standalone mode reports the storage directory plus its pool observation. Cluster mode reports the configured drive paths.
+
+## Cluster Health
+
+```typescript
+const clusterHealth = await storage.getClusterHealth();
+
+if (!clusterHealth.enabled) {
+  console.log('Cluster mode is disabled');
+} else {
+  console.log(clusterHealth.nodeId, clusterHealth.quorumHealthy);
+  console.log(clusterHealth.peers);
+  console.log(clusterHealth.drives);
+}
+```
+
+```typescript
+interface IClusterHealth {
+  enabled: boolean;
+  nodeId?: string;
+  quorumHealthy?: boolean;
+  majorityHealthy?: boolean;
+  peers?: IClusterPeerHealth[];
+  drives?: IClusterDriveHealth[];
+  erasure?: IClusterErasureHealth;
+  repairs?: IClusterRepairHealth;
+}
+```
+
+- `getClusterHealth()` is served by the Rust core. The TypeScript wrapper does not infer values from static config.
+- Standalone mode returns `{ enabled: false }`.
+- Peer status is the local node's current view of cluster membership and heartbeats, so it is best-effort and may lag real network state.
+- Drive health is based on live native probe checks on the configured local drive paths. Capacity values are cheap filesystem snapshots.
+- `quorumHealthy` means the local node currently sees majority quorum and enough available placements in every erasure set to satisfy the configured write quorum.
+- Repair fields expose the background healer's currently available runtime state. They are best-effort and limited to what the engine tracks today, such as whether a scan is active, the last completed run, and the last error.
+
+## Usage with AWS SDK v3
+
+```typescript
+import { S3Client, PutObjectCommand, GetObjectCommand, DeleteObjectCommand } from '@aws-sdk/client-s3';
+
+const descriptor = await storage.getStorageDescriptor();
+
+const client = new S3Client({
+  endpoint: `http://${descriptor.endpoint}:${descriptor.port}`,
+  region: 'us-east-1',
+  credentials: {
+    accessKeyId: descriptor.accessKey,
+    secretAccessKey: descriptor.accessSecret,
+  },
+  forcePathStyle: true,  // Required for path-style access
+});
+
+// Upload
+await client.send(new PutObjectCommand({
+  Bucket: 'my-bucket',
+  Key: 'hello.txt',
+  Body: 'Hello, Storage!',
+  ContentType: 'text/plain',
+}));
+
+// Download
+const { Body } = await client.send(new GetObjectCommand({
+  Bucket: 'my-bucket',
+  Key: 'hello.txt',
+}));
+const content = await Body.transformToString(); // "Hello, Storage!"
+
+// Delete
+await client.send(new DeleteObjectCommand({
+  Bucket: 'my-bucket',
+  Key: 'hello.txt',
+}));
+```
+
+## Usage with SmartBucket
+
+```typescript
+import { SmartBucket } from '@push.rocks/smartbucket';
+
+const smartbucket = new SmartBucket(await storage.getStorageDescriptor());
+const bucket = await smartbucket.createBucket('my-bucket');
+const dir = await bucket.getBaseDirectory();
+
+// Upload
+await dir.fastPut({ path: 'docs/readme.txt', contents: 'Hello!' });
+
+// Download
+const content = await dir.fastGet('docs/readme.txt');
+
+// List
+const files = await dir.listFiles();
+```
+
+Standalone ObjectStorage honors `If-None-Match: *` on both `PutObject` and
+`CompleteMultipartUpload`, publishes through an atomic no-replace operation,
+and implements bounded `ListParts` responses so exact-streaming clients can
+verify multipart abort cleanup. Conditional create-only writes and active
+`ListParts` fail closed in cluster mode until those semantics have complete
+durable cluster metadata. Descriptor-confined standalone startup reclaims
+abandoned private object materializations from interrupted create-only, copy,
+or multipart publication without removing an already published hard link.
+
+Standalone `DeleteObject` also honors `If-Match: *` and comma-separated strong
+ETags. It compares the current ETag and deletes inside the same bucket mutation
+gate used by PUT, COPY, and multipart completion. A mismatch returns the S3
+`PreconditionFailed` response without changing object or runtime-stat state; an
+absent conditional target returns `NoSuchKey`. This supplies the conditional
+delete semantics used by SmartBucket's verified exact-path purge capability.
+Conditional deletion returns `NotImplemented` in cluster mode until the
+precondition can be enforced across cluster nodes.
+
+AWS SDK streaming `PutObject` and `UploadPart` requests are decoded before
+storage. ObjectStorage verifies the declared decoded length and CRC32 trailer
+for `STREAMING-UNSIGNED-PAYLOAD-TRAILER`, and verifies the complete chained
+signature sequence for `STREAMING-AWS4-HMAC-SHA256-PAYLOAD`. The
+`aws-chunked` transport encoding is not retained as object metadata. Every
+`aws-chunked` request requires a verified SigV4 envelope, even when
+`auth.enabled` is `false`; ordinary non-streaming SDK requests retain the
+configured auth-disabled behavior. Unsupported, malformed, truncated,
+checksum-invalid, or signature-invalid bodies fail closed. Standalone
+overwrites use private same-filesystem staging and are atomically published
+only after terminal verification succeeds. Body verification failures and
+cancellation before publication preserve the previous object.
+Decoder memory is globally bounded, and `limits.requestTimeout` (milliseconds)
+bounds inactivity while waiting for each incoming body frame; timeout or
+memory-pressure failures publish no partial object.
+
+## Multipart Uploads
+
+For files larger than 5 MB, use multipart uploads. objectstorage handles them
+with **bounded-memory streaming I/O**. AWS streaming transport chunks are
+decoded through per-chunk buffers of at most 16 MiB under a shared 64 MiB
+decoder budget before the part data is written. In cluster mode, each part is
+independently erasure-coded and distributed.
+
+```typescript
+import {
+  CreateMultipartUploadCommand,
+  UploadPartCommand,
+  CompleteMultipartUploadCommand,
+} from '@aws-sdk/client-s3';
+
+// 1. Initiate
+const { UploadId } = await client.send(new CreateMultipartUploadCommand({
+  Bucket: 'my-bucket',
+  Key: 'large-file.bin',
+}));
+
+// 2. Upload parts
+const parts = [];
+for (let i = 0; i < chunks.length; i++) {
+  const { ETag } = await client.send(new UploadPartCommand({
+    Bucket: 'my-bucket',
+    Key: 'large-file.bin',
+    UploadId,
+    PartNumber: i + 1,
+    Body: chunks[i],
+  }));
+  parts.push({ PartNumber: i + 1, ETag });
+}
+
+// 3. Complete
+await client.send(new CompleteMultipartUploadCommand({
+  Bucket: 'my-bucket',
+  Key: 'large-file.bin',
+  UploadId,
+  MultipartUpload: { Parts: parts },
+}));
+```
+
+## Bucket Policies
+
+objectstorage supports AWS-style bucket policies for fine-grained access control. Policies use the same IAM JSON format as real S3 — so you can develop and test your policy logic locally before deploying.
+
+When `auth.enabled` is `true`, the auth pipeline works as follows:
+1. **Authenticate** — verify the AWS SigV4 signature (anonymous requests skip this step)
+2. **Authorize** — evaluate bucket policies against the request action, resource, and caller identity
+3. **Default** — authenticated users get full access; anonymous requests are denied unless a policy explicitly allows them
+
+### Setting a Bucket Policy
+
+```typescript
+import { PutBucketPolicyCommand } from '@aws-sdk/client-s3';
+
+// Allow anonymous read access to all objects in a bucket
+await client.send(new PutBucketPolicyCommand({
+  Bucket: 'public-assets',
+  Policy: JSON.stringify({
+    Version: '2012-10-17',
+    Statement: [{
+      Sid: 'PublicRead',
+      Effect: 'Allow',
+      Principal: '*',
+      Action: ['s3:GetObject'],
+      Resource: ['arn:aws:s3:::public-assets/*'],
+    }],
+  }),
+}));
+```
+
+### Policy Features
+
+- **Effect**: `Allow` and `Deny` (explicit Deny always wins)
+- **Principal**: `"*"` (everyone) or `{ "AWS": ["arn:..."] }` for specific identities
+- **Action**: IAM-style actions like `s3:GetObject`, `s3:PutObject`, `s3:*`, or prefix wildcards like `s3:Get*`
+- **Resource**: ARN patterns with `*` and `?` wildcards (e.g. `arn:aws:s3:::my-bucket/*`)
+- **Persistence**: Policies survive server restarts — stored as JSON on disk alongside your data
+
+### Policy CRUD Operations
+
+| Operation | AWS SDK Command | HTTP |
+|-----------|----------------|------|
+| Get policy | `GetBucketPolicyCommand` | `GET /{bucket}?policy` |
+| Set policy | `PutBucketPolicyCommand` | `PUT /{bucket}?policy` |
+| Delete policy | `DeleteBucketPolicyCommand` | `DELETE /{bucket}?policy` |
+
+Deleting a bucket automatically removes its associated policy.
+
+## Clustering Deep Dive 🔗
+
+objectstorage can run as a distributed storage cluster where multiple nodes cooperate to store and retrieve data with built-in redundancy.
+
+### How It Works
+
+```
+Client ──HTTP PUT──▶ Node A (coordinator)
+                       │
+                       ├─ Split object into 4 MB chunks
+                       ├─ Erasure-code each chunk (4 data + 2 parity = 6 shards)
+                       │
+                       ├──QUIC──▶ Node B (shard writes)
+                       ├──QUIC──▶ Node C (shard writes)
+                       └─ Local disk (shard writes)
+```
+
+1. **Any node can coordinate** — the client connects to any cluster member
+2. **Objects are chunked** — large objects split into fixed-size pieces (default 4 MB)
+3. **Each chunk is erasure-coded** — Reed-Solomon produces k data + m parity shards
+4. **Shards are distributed** — placed across different nodes and drives for fault isolation
+5. **Quorum guarantees consistency** — writes need k+1 acks, reads need k shards
+
+### Cluster Root and Upgrade Requirements
+
+Cluster mode never creates configured control or drive roots. Before startup,
+create `storage.directory` and every `cluster.drives.paths` entry, make them
+owned by the ObjectStorage process user, and remove group/world write permission.
+Every configured root must be a real directory with no symlink path component,
+and every drive root must resolve to a distinct device/inode identity. The
+control root may be distinct or may be the same identity as one drive root; it
+cannot make two drive entries aliases of one another. Every root must support
+the filesystem durability and locking semantics validated at startup.
+ObjectStorage fails closed before accepting traffic when a root does not meet
+these requirements.
+
+This release migrates clustered control records, manifests, and shards to
+descriptor-anchored, digest-addressed formats with durable tombstones. Legacy
+records are migrated in bounded steps and the new records become authoritative.
+Before a manifest listing can issue a continuation token, ObjectStorage drains
+all remaining legacy manifests through batches retaining at most 32 MiB, so a
+later migration cannot introduce a key behind an issued token. Background
+healing can stop that drain between committed record migrations during
+shutdown.
+Upgrade every member during one coordinated maintenance window. Once any member
+has written or migrated the new state, do not restart an older binary and do not
+run a mixed-version cluster. Back up the cluster roots before upgrading; rollback
+requires restoring those roots from the pre-upgrade backup.
+
+### Erasure Coding
+
+With the default `4+2` configuration:
+- Storage overhead: **33%** (vs. 200% for 3x replication)
+- Fault tolerance: **any 2 drives/nodes can fail** simultaneously
+- Read efficiency: only **4 of 6 shards** needed to reconstruct data
+
+| Config | Total Shards | Overhead | Tolerance | Min Nodes |
+|--------|-------------|----------|-----------|-----------|
+| 4+2 | 6 | 33% | 2 failures | 3 |
+| 6+3 | 9 | 50% | 3 failures | 5 |
+| 2+1 | 3 | 50% | 1 failure | 2 |
+
+### QUIC Transport
+
+Inter-node communication uses [QUIC](https://en.wikipedia.org/wiki/QUIC) via the `quinn` library:
+- 🔒 **Built-in TLS** — self-signed certs auto-generated at cluster init
+- 🔀 **Multiplexed streams** — concurrent shard transfers without head-of-line blocking
+- ⚡ **Connection pooling** — persistent connections to peer nodes
+- 🌊 **Natural backpressure** — QUIC flow control prevents overloading slow peers
+
+### Cluster Membership
+
+- **Static seed nodes** — initial cluster defined in config
+- **Runtime join** — new nodes can join a running cluster
+- **Heartbeat monitoring** — every 5s (configurable), with suspect/offline detection
+- **Split-brain prevention** — nodes only mark peers offline when they have majority
+
+### Self-Healing
+
+A background scanner periodically (default: every 24h):
+1. Checks shard checksums (CRC32C) for bit-rot detection
+2. Identifies shards on offline nodes
+3. Reconstructs missing shards from remaining data using Reed-Solomon
+4. Places healed shards on healthy drives
+
+Healing runs at low priority to avoid impacting foreground I/O.
+
+### Erasure Set Formation
+
+Drives are organized into fixed **erasure sets** at cluster initialization:
+
+```
+3 nodes × 4 drives each = 12 drives total
+With 6-shard erasure sets → 2 erasure sets
+
+Set 0: Node1-Disk0, Node2-Disk0, Node3-Disk0, Node1-Disk1, Node2-Disk1, Node3-Disk1
+Set 1: Node1-Disk2, Node2-Disk2, Node3-Disk2, Node1-Disk3, Node2-Disk3, Node3-Disk3
+```
+
+Drives are interleaved across nodes for maximum fault isolation. New nodes form new erasure sets — existing data is never rebalanced.
+
+## Testing Integration
+
+```typescript
+import { ObjectStorage } from '@lossless.org/objectstorage';
+import { tap, expect } from '@git.zone/tstest/tapbundle';
+
+let storage: ObjectStorage;
+
+tap.test('setup', async () => {
+  storage = await ObjectStorage.createAndStart({
+    server: { port: 4567, silent: true },
+    storage: { cleanSlate: true },
+  });
+});
+
+tap.test('should store and retrieve objects', async () => {
+  await storage.createBucket('test');
+  // ... your test logic using AWS SDK or SmartBucket
+});
+
+tap.test('teardown', async () => {
+  await storage.stop();
+});
+
+export default tap.start();
+```
+
+## API Reference
+
+### `ObjectStorage` Class
+
+#### `static createAndStart(config?: IObjectStorageConfig): Promise<ObjectStorage>`
+
+Create and start a server in one call.
+
+#### `static probeStoragePool(pool: IMountedStoragePoolConfig): Promise<IStoragePoolObservation>`
+
+Inspect an exact host-mounted NFS or SMB pool without writing to it or starting
+the S3 listener. The observation reports the active mount source, filesystem
+type, capacity/tuning metadata, stability checks, and current production
+eligibility.
+
+#### `start(): Promise<void>`
+
+Spawn the Rust binary and start the HTTP server. Concurrent starts join the same
+attempt; starting an already running instance succeeds without restarting it.
+A start is rejected while shutdown or unconfirmed startup cleanup remains
+pending. After completed shutdown, an explicit start begins a new lifecycle.
+
+#### `stop(): Promise<void>`
+
+Gracefully stop the server and then terminate the Rust process. In every server
+mode, an incomplete 30-second drain rejects this call without terminating the
+process; call `stop()` again after the blocked operation recovers. A terminal
+shutdown error is reported only after the completed Rust server has released
+its task and resource ownership; the wrapper terminates the management process
+before rejecting with that error.
+
+Concurrent stops join the same attempt, including any admitted startup. A stop
+after failed startup joins any remaining bridge cleanup without sending a stop
+command to an already terminated process. Repeated completed stops replay the
+same outcome, including terminal shutdown errors. Incomplete drain or process
+termination remains retryable; completed stop commands are retained so a
+termination retry cannot issue a duplicate command. A successful stop therefore
+confirms that the instance no longer owns its listener, work, or process; no
+health polling is required before removing a disposable storage root.
+
+The Rust management process reads stdin on a dedicated task. Ordinary commands
+enter a bounded nonblocking queue and continue to execute one at a time, while
+`stop` uses an independent retained lifecycle signal plus bounded waiters. The
+first stop intent immediately closes management and HTTP admission and signals
+HTTP, multipart, and clustered background producers even when an active mounted
+filesystem operation is blocked. Already queued ordinary commands are rejected
+as stopping. A stop waiter that reaches its deadline receives one incomplete
+error and is never sent a later completion; a retry uses a new request ID and
+receives the retained terminal result when ownership completes.
+
+IPC EOF requests the same graceful shutdown. If an owned blocked operation
+cannot finish by the EOF ownership deadline, the Rust process exits nonzero
+instead of returning through runtime teardown that could wait forever on a
+blocked native filesystem worker. Output is serialized through a dedicated
+writer with a 144 MiB per-line ceiling, a 160 MiB owned queue budget, and a
+five-second Unix stdout write deadline; writer failure or sustained stdout
+backpressure fails closed rather than growing memory without bound.
+
+#### `getStoragePoolDiagnostic(): Promise<IStoragePoolObservation | null>`
+
+Return the active pool observation. For a poisoned mounted pool, this uses
+cached activation evidence and remains available without touching the storage
+filesystem. It returns `null` before the Rust server has started.
+
+#### `createBucket(name: string): Promise<{ name: string }>`
+
+Create a storage bucket.
+
+#### `createBucketTenant(options): Promise<IBucketTenantDescriptor>`
+
+Create a bucket tenant with a generated or supplied scoped credential. Options: `{ bucketName, accessKeyId?, secretAccessKey?, region? }`.
+
+#### `ensureBucketTenant(options): Promise<IBucketTenantDescriptor>`
+
+Atomically create an absent bucket and exact scoped credential, resume an exact
+orphan credential, or rotate the credential for an existing exact-owned bucket.
+Foreign or ambiguous ownership is rejected before mutation. Options:
+`{ bucketName, accessKeyId, secretAccessKey, region?, fence? }`. Once a bucket
+has durable fence history, supply `fence`; the returned descriptor then includes
+the durable `resourceFence` receipt.
+
+#### `deleteBucketTenant(options): Promise<void>`
+
+Revoke a tenant credential or delete a bucket that still has tenant credentials. Options: `{ bucketName, accessKeyId? }`.
+
+#### `rotateBucketTenantCredentials(options): Promise<IBucketTenantDescriptor>`
+
+Replace the scoped credential for a bucket tenant. Options: `{ bucketName, accessKeyId?, secretAccessKey?, region? }`.
+
+#### `listBucketTenants(): Promise<IBucketTenantMetadata[]>`
+
+List scoped tenant credential metadata without returning secrets.
+
+#### `getBucketTenantDescriptor(options): Promise<IBucketTenantDescriptor>`
+
+Return endpoint, port, region, bucket, access key, secret key, SSL flag, legacy descriptor fields, and S3/AWS env values for the bucket tenant.
+
+#### `getBucketMigrationCapability(): Promise<IObjectStorageBucketMigrationCapability>`
+
+Return standalone provider identity and receipt-key evidence plus source seal,
+destination publication, and controller-retried source-cleanup capabilities.
+Cluster mode reports this migration metadata surface as unsupported.
+
+#### `createSourceMigration(options): Promise<IObjectStorageMigrationTransitionReceipt>`
+
+Create durable source migration ownership for an existing non-retained bucket.
+`sourceCleanupAccessKeyId` is required and must name the bucket's one exact
+scoped credential; its hash is permanently bound to the migration.
+
+#### `createDestinationMigration(options): Promise<IObjectStorageMigrationTransitionReceipt>`
+
+Create durable destination migration ownership for an existing bucket.
+
+#### `transitionSourceMigration(options): Promise<IObjectStorageMigrationTransitionReceipt>`
+
+Advance the source across an allowed phase edge. Sealing introduces the
+manifest digest; releasing requires the exact signed destination-abort receipt.
+
+#### `transitionDestinationMigration(options): Promise<IObjectStorageMigrationTransitionReceipt>`
+
+Advance the destination across an allowed phase edge. Verification introduces
+the manifest digest and terminal publication produces a signed receipt.
+
+#### `cleanupSourceMigration(options): Promise<IObjectStorageSourceMigrationCleanupReceipt>`
+
+Permanently delete a retired source after verifying the exact signed
+destination-published receipt. The request includes the migration binding,
+cleanup fence token, source capability, retired receipt hash, published receipt,
+exact resource fence, and exact source `accessKeyId`. Exact retries return the
+same durable cleanup receipt.
+
+#### `inspectBucketMigration(options): Promise<IObjectStorageMigrationInspection | null>`
+
+Return durable migration, terminal-receipt, and source-cleanup state for one
+`migrationId`.
+
+#### `exportBucket(options): Promise<IBucketExport>`
+
+Export one bucket's objects and metadata into a `smartstorage.bucket.v1` JSON
+object, subject to the bounded legacy management-export limits documented
+above.
+
+#### `importBucket(options): Promise<IObjectStorageBucketMutationResult>`
+
+Import a `smartstorage.bucket.v1` JSON object into the target bucket after
+validating object size and MD5. An exact fenced import binds the replacement to
+the bucket's exclusive credential. Options:
+`{ bucketName, source, accessKeyId?, fence?, holdPublication? }`. Setting
+`holdPublication: true` requires a fence and returns
+`resourcePublicationBarrier`; omitted or `false` preserves immediate behavior.
+
+#### `deleteBucket(options): Promise<IObjectStorageBucketMutationResult>`
+
+Delete a fenced bucket. Supplying `accessKeyId` also durably revokes that
+exact tenant credential and deletes the policy; omitting it preserves the
+legacy credential/policy behavior. Options:
+`{ bucketName, accessKeyId?, fence, holdPublication? }`. Setting
+`holdPublication: true` returns a durable `resourcePublicationBarrier`; omitted
+or `false` preserves immediate behavior.
+
+#### `commitResourcePublication(options): Promise<IObjectStorageResourcePublicationReleaseReceipt>`
+
+Release a durable write-publication hold after the caller has persisted and
+published the exact provider receipt. Options:
+`{ barrier, resourceFenceReceipt }`. The operation is exact-match and idempotent.
+
+#### `getResourceFencingCapability(): Promise<IObjectStorageResourceFencingCapability>`
+
+Report resource-fencing support, drain requirements, publication-hold version,
+whether publication holds are supported, and whether the provider must perform
+an additional external drain.
+
+#### `getStorageDescriptor(options?): Promise<IS3Descriptor>`
+
+Get connection details for S3-compatible clients. Returns:
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `endpoint` | `string` | Server hostname (`localhost` by default) |
+| `port` | `number` | Server port |
+| `accessKey` | `string` | Access key from first configured credential |
+| `accessSecret` | `string` | Secret key from first configured credential |
+| `useSsl` | `boolean` | Always `false` (plain HTTP) |
+
+#### `getStorageStats(): Promise<IStorageStats>`
+
+Read cached logical bucket and object totals from the Rust runtime without issuing S3 list calls.
+
+#### `listBucketSummaries(): Promise<IBucketSummary[]>`
+
+Get per-bucket logical object counts and total payload sizes.
+
+#### `listCredentials(): Promise<IStorageCredentialMetadata[]>`
+
+Return metadata for the currently active runtime credential set without `secretAccessKey` values.
+
+#### `replaceCredentials(credentials: IStorageCredential[]): Promise<void>`
+
+Atomically replace the active runtime credential set without restarting the server.
+
+#### `getClusterHealth(): Promise<IClusterHealth>`
+
+Read the Rust core's current cluster, drive, quorum, and repair health snapshot. Standalone mode returns `{ enabled: false }`.
+
+#### `getHealth(): Promise<IObjectStorageHealth>`
+
+Return running state, storage directory, per-location pool observations, auth
+state, credential counts, bucket count, object count, total bytes, cluster
+health, resource-fencing capability, and publication-hold capability fields.
+
+#### `getMetrics(): Promise<IObjectStorageMetrics>`
+
+Return numeric metrics plus a Prometheus text snippet for operational scraping.
+
+## Architecture
+
+objectstorage uses a **hybrid Rust + TypeScript** architecture:
+
+```
+┌──────────────────────────────────────────────┐
+│  Your Code (AWS SDK, SmartBucket, etc.)       │
+│  ↕ HTTP (localhost:3000)                     │
+├──────────────────────────────────────────────┤
+│  ruststorage binary (Rust)                    │
+│  ├─ hyper 1.x HTTP server                   │
+│  ├─ S3 path-style routing                   │
+│  ├─ StorageBackend (Standalone or Clustered) │
+│  │   ├─ FileStore (single-node mode)        │
+│  │   └─ DistributedStore (cluster mode)     │
+│  │       ├─ ErasureCoder (Reed-Solomon)     │
+│  │       ├─ ShardStore (per-drive storage)  │
+│  │       ├─ QuicTransport (quinn)           │
+│  │       ├─ ClusterState & Membership       │
+│  │       └─ HealingService                  │
+│  ├─ SigV4 auth + policy engine              │
+│  ├─ CORS middleware                          │
+│  └─ S3 XML response builder                 │
+├──────────────────────────────────────────────┤
+│  TypeScript (thin IPC wrapper)               │
+│  ├─ ObjectStorage class                       │
+│  ├─ RustBridge (stdin/stdout JSON IPC)       │
+│  └─ Config & S3 descriptor                  │
+└──────────────────────────────────────────────┘
+```
+
+**Why Rust?** The original TypeScript implementation had critical perf issues: OOM on multipart uploads (parts buffered in memory), double stream copying, file descriptor leaks on HEAD requests, full-file reads for range requests, and no backpressure. The Rust binary solves these with bounded-memory streaming I/O, backpressure, and direct `seek()` for range requests.
+
+**IPC Protocol:** TypeScript communicates with the `ruststorage` binary over newline-delimited JSON via stdin/stdout. The current management commands are `probeStoragePool`, `getStoragePoolDiagnostic`, `start`, `stop`, `createBucket`, `createBucketTenant`, `ensureBucketTenant`, `deleteBucketTenant`, `rotateBucketTenantCredentials`, `listBucketTenants`, `getBucketTenantCredential`, `getBucketRetentionCapability`, `getBucketRetentionReceipt`, `getBucketMigrationCapability`, `createSourceMigration`, `createDestinationMigration`, `transitionSourceMigration`, `transitionDestinationMigration`, `cleanupSourceMigration`, `inspectBucketMigration`, `exportBucket`, `importBucket`, `deleteBucket`, `commitResourcePublication`, `getResourceFencingCapability`, `getStorageStats`, `listBucketSummaries`, `listCredentials`, `replaceCredentials`, and `getClusterHealth`. `getStoragePoolDiagnostic` reads a weak, cached diagnostic source and remains responsive while another management command owns the server; it does not retain the mounted root after server ownership completes.
+
+### S3-Compatible Operations
+
+Bucket names are validated consistently at HTTP, management, policy, and
+storage boundaries: they must be 3-63 lowercase letters, digits, dots, or
+hyphens, begin and end with a letter or digit, contain no consecutive dots,
+and not be formatted as an IPv4 address.
+
+Object keys remain S3-compatible UTF-8 strings of up to 1,024 bytes, including
+leading, repeated, embedded, and trailing `/` characters. On disk, every new
+object key is encoded below the reserved `.smartstorage-objects-v2` bucket
+directory as canonical, prefix-free lowercase-hex frames. Frames and fixed
+payload/sidecar names stay within the common 255-byte `NAME_MAX`, remain safe on
+CIFS, and listings return the exact original key.
+
+Existing v7 identity-layout objects remain readable on local and NFS storage
+when their raw path can be reconstructed without traversal. Their first
+mutation stages and publishes the canonical v2 object set before removing the
+legacy payload and sidecars. The migration uses descriptor-confined,
+digest-bound durable state and reconciles every preparation, publication, and
+partial-cleanup crash on startup or next access. Canonical and legacy payloads
+for the same key without that exact owned state, noncanonical entries below the
+reserved v2 root, and unsafe legacy aliases on case-folding CIFS fail closed
+instead of choosing an ambiguous object.
+
+The standalone v2 object-layout upgrade is one-way. After the first canonical
+v2 object write or legacy-object migration in a standalone storage root,
+downgrading that root to `@push.rocks/smartstorage` 7.0.0 or earlier is unsupported because
+those versions cannot read the canonical v2 objects.
+
+| Operation | Method | Path |
+|-----------|--------|------|
+| ListBuckets | `GET /` | |
+| CreateBucket | `PUT /{bucket}` | |
+| DeleteBucket | `DELETE /{bucket}` | |
+| HeadBucket | `HEAD /{bucket}` | |
+| ListObjects (v1/v2) | `GET /{bucket}` | `?list-type=2` for v2 |
+| PutObject | `PUT /{bucket}/{key}` | |
+| GetObject | `GET /{bucket}/{key}` | Supports `Range` header |
+| HeadObject | `HEAD /{bucket}/{key}` | |
+| DeleteObject | `DELETE /{bucket}/{key}` | Standalone `If-Match: *` or strong ETag list |
+| DeleteObjects | `POST /{bucket}?delete` | Up to 1,000 keys with CRC32 or Content-MD5 integrity |
+| CopyObject | `PUT /{bucket}/{key}` | `x-amz-copy-source` header |
+| InitiateMultipartUpload | `POST /{bucket}/{key}?uploads` | |
+| UploadPart | `PUT /{bucket}/{key}?partNumber&uploadId` | |
+| ListParts | `GET /{bucket}/{key}?uploadId` | Optional `max-parts` and `part-number-marker` |
+| CompleteMultipartUpload | `POST /{bucket}/{key}?uploadId` | |
+| AbortMultipartUpload | `DELETE /{bucket}/{key}?uploadId` | |
+| ListMultipartUploads | `GET /{bucket}?uploads` | |
+| GetBucketPolicy | `GET /{bucket}?policy` | |
+| PutBucketPolicy | `PUT /{bucket}?policy` | |
+| DeleteBucketPolicy | `DELETE /{bucket}?policy` | |
+
+### On-Disk Format
+
+**Standalone mode:**
+```
+{storage.directory}/
+  {bucket}/
+    .smartstorage-objects-v2/
+      {whole-key-hex-frames}/
+        object._storage_object_v2                # Object data; completed multipart objects include
+                                                 # an atomic payload/ETag/metadata trailer
+        object._storage_object_v2.metadata.json  # Legacy/direct-write metadata sidecar
+        object._storage_object_v2.md5            # Legacy/direct-write cached MD5 hash
+  .multipart/
+    {upload-id}/
+      metadata.json                      # Upload metadata
+      part-1, part-2, ...               # Part data files
+  .policies/
+    {bucket}.policy.json                 # Bucket policy (IAM JSON format)
+```
+
+**Cluster mode:**
+```
+{storage.directory}/
+  .buckets/
+    v2/{bucket-address}.bucket.json       # Active/tombstoned bucket state
+    .policies/{bucket}.policy.json        # Bucket policies
+  .manifests/
+    v2/{bucket-prefix}/{bucket-address}/{object-prefix}/
+      {object-address}.state.json
+                                         # Active/tombstoned object manifest
+    .multipart/{upload-id}/               # Descriptor-anchored multipart state
+    .get-tmp/                             # Unlinked GET reconstruction files
+  .smartstorage/cluster/
+    identity.json                         # Node and cluster identity
+    topology.json                         # Last durable topology
+    generation-inventory/{address}.json   # Deferred physical-generation cleanup
+
+{drive_path}/.smartstorage/data/
+  v2/{bucket-prefix}/{bucket-address}/{key-prefix}/{generation-address}/
+    {shard-address}.shard                 # Immutable erasure-coded shard record
+    {shard-address}.tombstone             # Crash-safe deletion authority
+```
+
+Superseded physical generations are queued with their exact shard placements.
+Cluster reads are bounded, cleanup waits a 24-hour reader grace period, and the
+hourly cleanup worker purges both shard records and tombstones. Multipart
+uploads older than `multipart.expirationDays`, plus committed uploads whose
+first cleanup attempt failed, are retried during bounded cleanup passes
+scheduled every `multipart.cleanupIntervalMinutes`; large registries are
+visited page by page across successive passes. Generation cleanup retains at
+most 32 MiB of serialized records per page, multipart session listings process
+documents one at a time, policy startup loading is bounded and streaming, and
+both workers observe shutdown within an active pass. Both multipart settings
+must be greater than zero; ObjectStorage rejects invalid values during startup.
+Shutdown never aborts or detaches an active multipart descriptor worker; it
+waits for the worker under the same retryable bounded drain used by mounted
+HTTP requests.
+Multipart admission is bounded at 100,000 registry entries per storage
+process, including cleanup-pending and unreadable entries. Cleanup only removes
+descriptor-verified expired uploads and cleanup-marked committed entries;
+unreadable or incomplete descriptors fail closed and require operator repair
+or removal after their bucket and clustered shard ownership has been
+established.
+
+## Related Packages
+
+- [`@push.rocks/smartbucket`](https://code.foss.global/push.rocks/smartbucket) — High-level S3-compatible abstraction layer
+- [`@push.rocks/smartrust`](https://code.foss.global/push.rocks/smartrust) — TypeScript ↔ Rust IPC bridge
+- [`@git.zone/tsrust`](https://code.foss.global/git.zone/tsrust) — Rust cross-compilation for npm packages
+
+## License and Legal Information
+
+This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the [license](./license.md) file.
+
+**Please note:** The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.
+
+### Trademarks
+
+This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.
+
+Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.
+
+### Company Information
+
+Task Venture Capital GmbH
+Registered at District Court Bremen HRB 35230 HB, Germany
+
+For any legal inquiries or further information, please contact us via email at hello@task.vc.
+
+By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.