@cloudbitmaps/s3 0.22.0 → 0.23.0

package/README.md

@@ -14,6 +14,8 @@ service to run.
 pnpm add @cloudbitmaps/roaring @cloudbitmaps/s3
 ```
 
+Install the same version of every `@cloudbitmaps/*` package: they release together, and each pins the exact version of `core` it was built with.
+
 `roaring` needs its one install script to fetch its native binary, and newer clients run a dependency's script only if you allow it. Allow it before you install:
 
 - **npm 12:** add `{ "allowScripts": { "roaring": true } }` to your `package.json`. Without it the install exits 0 and the package throws at `import`. npm 11 runs the script but warns until you allow it the same way.
@@ -28,6 +30,7 @@ pnpm add @cloudbitmaps/roaring @cloudbitmaps/s3
 
 ## Use
 
+<!-- typecheck -->
 ```ts
 import { CloudRoaring } from '@cloudbitmaps/roaring';
 import { S3Storage } from '@cloudbitmaps/s3';
@@ -49,12 +52,17 @@ It builds its own client from your usual AWS credentials. Any other key is refus
 | `client` | your own `S3Client`; it carries its own region, endpoint and credentials |
 | `region`, `endpoint`, `pathStyle`, `credentials` | build a client for you, such as one for MinIO; refused beside `client` |
 | `maxSockets` | most sockets the client you do not pass opens at once, for `https` and plain `http` alike; a positive integer, default 128 (the SDK's own is 50; if a future SDK changes its handler, the built client keeps the SDK's own limit); refused beside `client` |
-| `partBytes`, `maxObjectBytes` | multipart sizing: part size (default 8 MiB; the upload buffers one part at a time) and the largest object (about 80 GiB by default, up to S3's 5 TiB). A cap above 5 TiB, or a part above S3's 5 GiB part limit, is a `ValidationError` |
+| `partBytes`, `maxObjectBytes` | multipart sizing: part size (default 8 MiB; the upload buffers one part at a time and copies it into one body to send it, so it holds up to about twice the part size at a flush, 16 MiB at the default, derived from the driver's code) and the largest object (about 78 GiB by default, up to S3's 5 TiB). A cap above 5 TiB, or a part above S3's 5 GiB part limit, is a `ValidationError` |
 | `readTimeoutMs` | how long one read (a `GetObject` or `HeadObject`, its body included) may take before it throws `TransientError` and the store retries it; `0`, the default, sets no timeout. Writes are not timed |
 | `conditionalDelete` | whether the store applies `If-Match` on a `DeleteObject`: the registry then removes a deleted row rather than leaving a tombstone, and the storage half deletes an erasure's holder only while it is the object the erasure read, by its ETag. On by default when the host the client resolves is an AWS S3 host, however its endpoint was set (`endpoint`, `AWS_ENDPOINT_URL_S3`, `AWS_ENDPOINT_URL`, the shared config file); off for any other host: set it for an S3-compatible store only once you know it applies the header (MinIO ignores it) |
 
 ## Before production
 
+- **Run `await store.preflight()` once before the first load.** It reads the registry, writes one probe object, writes the
+  same key again (which must be refused), reads it back, lists it, deletes it and checks the lifecycle rules, with
+  `s3:ListBucket`, `s3:GetObject`, `s3:PutObject` and `s3:DeleteObject`, and `s3:GetLifecycleConfiguration` for the last. It does
+  not check `s3:AbortMultipartUpload`, the registry's `If-Match`, or a read after a delete, among others: [the full list](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/production.md#what-preflight-does-not-check). It
+  shows an SDK of 3.640.0 or older as a `write-once` problem, and cannot see one from 3.641.0 to 3.699.0, so the floor below stays yours to check. The probe sits at `<prefix>cbm.due.preflight/`, outside your application's namespaces: a grant scoped per namespace shows as a `write` problem, and as a `registry` problem too when the registry grant is scoped per namespace.
 - **The bucket must honor conditional writes**: `If-None-Match: *` for a write-once generation and `If-Match` for the
   pointer. AWS S3 does, and the test suite runs against MinIO. Another S3-compatible service that ignores the headers
   turns write-once into overwrite, so check yours before you depend on it. It must also be strongly consistent: a read
@@ -74,18 +82,26 @@ It builds its own client from your usual AWS credentials. Any other key is refus
   `registry/` prefix. An expired registry row takes the segment's pointer with it, and the token every publish is fenced
   on, so its generations look unreferenced; an expired generation makes a live segment unreadable
   ([bucket lifecycle](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/production.md#bucket-lifecycle)).
-- **A transient failure of a write throws `TransientError`, and the write may or may not have landed.** Re-run the
-  call, or check `store.generations(ref)`. The SDK's own retry is off for conditional writes, so a write that landed
-  is not reported as a conflict. The driver sends each registry write once; a load that gets no answer reads the row and
-  sends a fresh compare-and-swap from it, at most three times. A generation's object is sent again, up to three more
-  times, only after a `503` (`SlowDown` is the usual one); it carries a random id in its user metadata
-  (`x-amz-meta-cbwid`), so a first send that landed is told from another writer's object. A load whose row writes all go
-  unanswered throws `TransientError` and keeps its object. The waits before an object's re-sends use real timers: the
-  store's injected clock does not reach them, and `S3Storage` takes no clock, so a test that throttles one object
-  write three times waits about 3.5 s at most (the publish's waits before a fresh row write do use the store's
-  clock). **A bare `429`** (AWS S3 does not send one; some
-  S3-compatible services do) **is not retried and is not classified transient**: it surfaces as the SDK's own error, so a
-  layer of yours that keys on `TransientError` will not retry it.
+- **Check the bucket's lifecycle rules with `await backend.checkLifecycle()`.** It reads the configuration with one
+  request (bounded by `readTimeoutMs`, when set) and answers `ok`, `problem` (an enabled, untagged rule over your prefix that expires current objects or moves
+  them to `GLACIER`, `DEEP_ARCHIVE` or a class it does not know) or `unchecked`. It needs `s3:GetLifecycleConfiguration`, which a load never uses and
+  `store.preflight()`'s lifecycle step does: without it the answer is `unchecked`, naming the permission. A missing abort-incomplete-upload rule is
+  advice in `detail`, never a `problem`
+  ([check the lifecycle rules from code](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/production.md#check-the-lifecycle-rules-from-code)).
+- **A transient failure of a write throws `TransientError`, and the write may or may not have landed.** Re-run the call,
+  or check `store.generations(ref)`. The SDK's own retry is off for conditional writes, so a write that landed is not
+  reported as a conflict. The driver [sends each registry write
+  once](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/correctness.md#sdk-retry); a load that gets no
+  answer [reads the row and sends a fresh compare-and-swap from
+  it](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/correctness.md#row-write-settled), at most three
+  times. A generation's object is sent again, up to three more times, only after a `503` (`SlowDown` is the usual one) or a bare `429` ([why that is
+  safe](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/correctness.md#object-write-resent));
+  it carries a random id in its user metadata (`x-amz-meta-cbwid`), so a first send that landed is told from another
+  writer's object. A load whose row writes all go unanswered throws `TransientError` and keeps its object. The waits
+  before an object's re-sends use real timers: the store's injected clock does not reach them, and `S3Storage` takes no
+  clock, so a test that throttles one object write three times waits about 3.5 s at most (the publish's waits before a
+  fresh row write do use the store's clock). **A bare `429`** (AWS S3 does not send one; some S3-compatible services do)
+  **is a throttle, handled exactly as a `503` is**, and one that ends a call is a `TransientError`.
 - **Reads can be timed, and writes are not.** Nothing is timed unless you set `readTimeoutMs`. Set, a read that has
   not finished in that many ms throws `TransientError`, and the store runs it again; AWS's guidance is to retry a GET
   of under 512 KB after about 2 seconds. The timer covers the body as well, so a connection that sends its headers and

package/dist/backend.d.ts

@@ -1,5 +1,5 @@
 import { STORAGE_BACKEND } from '@cloudbitmaps/core/driver-kit';
-import type { IRegistryDriver, IStorageDriver, StorageBackend } from '@cloudbitmaps/core/driver-kit';
+import type { IRegistryDriver, IStorageDriver, LifecycleVerdict, StorageBackend } from '@cloudbitmaps/core/driver-kit';
 import { S3Client } from '@aws-sdk/client-s3';
 import type { IMetricsSink } from '@cloudbitmaps/core/driver-kit';
 /**
@@ -9,7 +9,7 @@ import type { IMetricsSink } from '@cloudbitmaps/core/driver-kit';
 export interface S3StorageOptions {
     /** Target bucket (must already exist). */
     readonly bucket: string;
-    /** Optional key prefix under which everything lives — generations and the registry alike. */
+    /** Optional key prefix under which everything lives — generations and the registry alike. A string, else `ValidationError`. */
     readonly prefix?: string;
     /**
      * A constructed client. Supply one for a credential chain the SDK cannot infer; otherwise one is built. Its retry
@@ -17,14 +17,27 @@ export interface S3StorageOptions {
      * configured to do: a registry row once, and a generation's object again only after a throttle, under its own backoff.
      */
     readonly client?: S3Client;
-    /** Region for the client built when `client` is absent (refused beside `client`). Falls back to the SDK's own resolution. */
+    /**
+     * Region for the client built when `client` is absent (refused beside `client`): a non-empty string, else
+     * `ValidationError`. Falls back to the SDK's own resolution.
+     */
     readonly region?: string;
-    /** Endpoint for an S3-compatible store (MinIO, Ceph, R2). Refused beside `client`, which carries its own. */
+    /**
+     * Endpoint for an S3-compatible store (MinIO, Ceph, R2): an http or https URL, since an empty one would send
+     * requests to AWS, else `ValidationError`. Refused beside `client`, which carries its own.
+     */
     readonly endpoint?: string;
-    /** Path-style addressing, which most S3-compatible stores require. Refused beside `client`, which carries its own. */
+    /**
+     * Path-style addressing, which most S3-compatible stores require: a boolean, else `ValidationError`. Refused beside
+     * `client`, which carries its own.
+     */
     readonly pathStyle?: boolean;
     /**
-     * Static credentials, for the S3-compatible stores that issue them (MinIO, Ceph, R2).
+     * Static credentials, for the S3-compatible stores that issue them (MinIO, Ceph, R2): non-empty `accessKeyId` and
+     * `secretAccessKey` strings and an optional `sessionToken`, else `ValidationError`, which never prints a value. No other
+     * key is taken except `$source`, `credentialScope`, `accountId` and `expiration`, which are tolerated and not read:
+     * the SDK stamps `$source` on a credentials object it is given, so one used for two stores carries it, and a credential
+     * provider's result carries the others. A provider function is not taken here; put it on an `S3Client` given as `client`.
      *
      * On AWS itself, leave this unset — the SDK's own chain (instance role, SSO, environment, profile) is what
      * you want, and hard-coding keys to reach it would be a downgrade. It exists because the alternative for a
@@ -48,7 +61,7 @@ export interface S3StorageOptions {
      */
     readonly maxSockets?: number;
     /**
-     * Largest object the backend will write and advertise. Default = `partBytes × 10,000` (≈ 80 GiB at the default
+     * Largest object the backend will write and advertise. Default = `partBytes × 10,000` (≈ 78 GiB at the default
      * 8 MiB part) — the honest ceiling reachable within S3's 10,000-part limit. Set it higher and `partBytes`
      * auto-grows so 10,000 parts still cover it (raising peak write memory to ~one part); up to the 5 TiB S3 max, and a
      * larger cap, or a `partBytes` above S3's 5 GiB part limit, is refused with `ValidationError`.
@@ -128,10 +141,32 @@ export declare class S3Storage implements StorageBackend {
     /** The client both halves share — built here unless one was supplied. */
     readonly client: S3Client;
     private readonly sockets;
+    private readonly bucket;
+    /** The key prefix every object sits under: `''`, or `trimmed/`. */
+    private readonly keyPrefix;
+    private readonly readTimeoutMs;
+    /**
+     * Builds the S3 backend over one bucket, which `bucket` names. Pass `client`, or `region`, `endpoint`, `pathStyle`,
+     * `credentials` and `maxSockets` to have one built, not both. `prefix` puts every object under a key prefix. Throws
+     * `ValidationError` for an unknown option, no bucket, a `now` that is not a function, a client setting beside
+     * `client`, a `maxSockets` (with no `client`) that is not a positive integer, a `partBytes` or `maxObjectBytes` that
+     * is not a positive integer, a `conditionalDelete` that is not a boolean, a `region` or `endpoint` that is empty or not a string, a `pathStyle` that
+     * is not a boolean, `credentials` that are not complete, or an invalid `prefix` or `readTimeoutMs`.
+     */
     constructor(options: S3StorageOptions);
     /**
      * Called by a store given a metrics sink. After the first read, a client whose socket pool is smaller than twice the
      * default window (64) sends the sink one `advisory` event, once; see the production guide's socket sizing.
      */
     attachMetrics(sink: IMetricsSink): void;
+    /**
+     * Reads the bucket's lifecycle configuration (one `GetBucketLifecycleConfiguration` request, plus the SDK's own
+     * retries, bounded by `readTimeoutMs` when it is set) and reports a rule that would expire, or move to GLACIER,
+     * DEEP_ARCHIVE or a class it does not know, the current objects under this store's prefix, generations and registry
+     * rows alike. Needs `s3:GetLifecycleConfiguration`, which a load never uses (`store.preflight()`'s
+     * lifecycle step does): without it the
+     * answer is `unchecked`, naming the permission. A missing rule that aborts incomplete multipart uploads is advice in
+     * `detail`, not a problem. Never rejects; a `detail` carries no credential. Rule reading: see the production guide.
+     */
+    checkLifecycle(): Promise<LifecycleVerdict>;
 }