@cloudbitmaps/s3 0.21.0 → 0.23.0

package/README.md

@@ -14,13 +14,23 @@ service to run.
 pnpm add @cloudbitmaps/roaring @cloudbitmaps/s3
 ```
 
-On npm 12 and pnpm 10 and later, allow `roaring`'s one install script first, or the install exits 0 and the package throws at `import`. Put this in your `package.json`: `{ "allowScripts": { "roaring": true }, "pnpm": { "onlyBuiltDependencies": ["roaring"] } }`. npm 11 runs the script but warns until you allow it the same way; pnpm 9 needs nothing extra. [Details](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/getting-started.md#cannot-find-module-buildreleaseroaringnode-after-a-successful-install).
+Install the same version of every `@cloudbitmaps/*` package: they release together, and each pins the exact version of `core` it was built with.
+
+`roaring` needs its one install script to fetch its native binary, and newer clients run a dependency's script only if you allow it. Allow it before you install:
+
+- **npm 12:** add `{ "allowScripts": { "roaring": true } }` to your `package.json`. Without it the install exits 0 and the package throws at `import`. npm 11 runs the script but warns until you allow it the same way.
+- **pnpm 11 and later:** put `allowBuilds: { roaring: true }` in a `pnpm-workspace.yaml` beside your `package.json`. Those versions no longer read the `package.json` `pnpm` key. Without it the install exits 1 with `ERR_PNPM_IGNORED_BUILDS`.
+- **pnpm 10:** add `{ "pnpm": { "onlyBuiltDependencies": ["roaring"] } }` to your `package.json`. Without it the install exits 0 and the package throws at `import`.
+- **pnpm 9:** nothing extra.
+
+[Details](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/getting-started.md#cannot-find-module-buildreleaseroaringnode-after-a-successful-install).
 
 `@aws-sdk/client-s3` (`>=3.700.0 <4`) is a real dependency of this package, so installing it is the whole step.
 `@cloudbitmaps/core`, the engine underneath, arrives with it and you never name it.
 
 ## Use
 
+<!-- typecheck -->
 ```ts
 import { CloudRoaring } from '@cloudbitmaps/roaring';
 import { S3Storage } from '@cloudbitmaps/s3';
@@ -42,15 +52,21 @@ It builds its own client from your usual AWS credentials. Any other key is refus
 | `client` | your own `S3Client`; it carries its own region, endpoint and credentials |
 | `region`, `endpoint`, `pathStyle`, `credentials` | build a client for you, such as one for MinIO; refused beside `client` |
 | `maxSockets` | most sockets the client you do not pass opens at once, for `https` and plain `http` alike; a positive integer, default 128 (the SDK's own is 50; if a future SDK changes its handler, the built client keeps the SDK's own limit); refused beside `client` |
-| `partBytes`, `maxObjectBytes` | multipart sizing: part size (default 8 MiB; the upload buffers one part at a time) and the largest object (about 80 GiB by default, up to S3's 5 TiB). A cap above 5 TiB, or a part above S3's 5 GiB part limit, is a `ValidationError` |
+| `partBytes`, `maxObjectBytes` | multipart sizing: part size (default 8 MiB; the upload buffers one part at a time and copies it into one body to send it, so it holds up to about twice the part size at a flush, 16 MiB at the default, derived from the driver's code) and the largest object (about 78 GiB by default, up to S3's 5 TiB). A cap above 5 TiB, or a part above S3's 5 GiB part limit, is a `ValidationError` |
 | `readTimeoutMs` | how long one read (a `GetObject` or `HeadObject`, its body included) may take before it throws `TransientError` and the store retries it; `0`, the default, sets no timeout. Writes are not timed |
 | `conditionalDelete` | whether the store applies `If-Match` on a `DeleteObject`: the registry then removes a deleted row rather than leaving a tombstone, and the storage half deletes an erasure's holder only while it is the object the erasure read, by its ETag. On by default when the host the client resolves is an AWS S3 host, however its endpoint was set (`endpoint`, `AWS_ENDPOINT_URL_S3`, `AWS_ENDPOINT_URL`, the shared config file); off for any other host: set it for an S3-compatible store only once you know it applies the header (MinIO ignores it) |
 
 ## Before production
 
+- **Run `await store.preflight()` once before the first load.** It reads the registry, writes one probe object, writes the
+  same key again (which must be refused), reads it back, lists it, deletes it and checks the lifecycle rules, with
+  `s3:ListBucket`, `s3:GetObject`, `s3:PutObject` and `s3:DeleteObject`, and `s3:GetLifecycleConfiguration` for the last. It does
+  not check `s3:AbortMultipartUpload`, the registry's `If-Match`, or a read after a delete, among others: [the full list](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/production.md#what-preflight-does-not-check). It
+  shows an SDK of 3.640.0 or older as a `write-once` problem, and cannot see one from 3.641.0 to 3.699.0, so the floor below stays yours to check. The probe sits at `<prefix>cbm.due.preflight/`, outside your application's namespaces: a grant scoped per namespace shows as a `write` problem, and as a `registry` problem too when the registry grant is scoped per namespace.
 - **The bucket must honor conditional writes**: `If-None-Match: *` for a write-once generation and `If-Match` for the
   pointer. AWS S3 does, and the test suite runs against MinIO. Another S3-compatible service that ignores the headers
-  turns write-once into overwrite, so check yours before you depend on it.
+  turns write-once into overwrite, so check yours before you depend on it. It must also be strongly consistent: a read
+  or a listing made after a write or a delete returns sees it. AWS S3 is.
 - **Use `@aws-sdk/client-s3` 3.700.0 or later** if you pin the SDK or pass your own `client`. This floor is a
   correctness floor, not a preference. An SDK sends only the conditional headers it models, and drops one it does
   not, without an error. `If-None-Match`, which write-once is built on, is modelled from 3.641.0: measured against
@@ -66,18 +82,26 @@ It builds its own client from your usual AWS credentials. Any other key is refus
   `registry/` prefix. An expired registry row takes the segment's pointer with it, and the token every publish is fenced
   on, so its generations look unreferenced; an expired generation makes a live segment unreadable
   ([bucket lifecycle](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/production.md#bucket-lifecycle)).
-- **A transient failure of a write throws `TransientError`, and the write may or may not have landed.** Re-run the
-  call, or check `store.generations(ref)`. The SDK's own retry is off for conditional writes, so a write that landed
-  is not reported as a conflict. The driver sends each registry write once; a load that gets no answer reads the row and
-  sends a fresh compare-and-swap from it, at most three times. A generation's object is sent again, up to three more
-  times, only after a `503` (`SlowDown` is the usual one); it carries a random id in its user metadata
-  (`x-amz-meta-cbwid`), so a first send that landed is told from another writer's object. A load whose row writes all go
-  unanswered throws `TransientError` and keeps its object. The waits before an object's re-sends use real timers: the
-  store's injected clock does not reach them, and `S3Storage` takes no clock, so a test that throttles one object
-  write three times waits about 3.5 s at most (the publish's waits before a fresh row write do use the store's
-  clock). **A bare `429`** (AWS S3 does not send one; some
-  S3-compatible services do) **is not retried and is not classified transient**: it surfaces as the SDK's own error, so a
-  layer of yours that keys on `TransientError` will not retry it.
+- **Check the bucket's lifecycle rules with `await backend.checkLifecycle()`.** It reads the configuration with one
+  request (bounded by `readTimeoutMs`, when set) and answers `ok`, `problem` (an enabled, untagged rule over your prefix that expires current objects or moves
+  them to `GLACIER`, `DEEP_ARCHIVE` or a class it does not know) or `unchecked`. It needs `s3:GetLifecycleConfiguration`, which a load never uses and
+  `store.preflight()`'s lifecycle step does: without it the answer is `unchecked`, naming the permission. A missing abort-incomplete-upload rule is
+  advice in `detail`, never a `problem`
+  ([check the lifecycle rules from code](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/production.md#check-the-lifecycle-rules-from-code)).
+- **A transient failure of a write throws `TransientError`, and the write may or may not have landed.** Re-run the call,
+  or check `store.generations(ref)`. The SDK's own retry is off for conditional writes, so a write that landed is not
+  reported as a conflict. The driver [sends each registry write
+  once](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/correctness.md#sdk-retry); a load that gets no
+  answer [reads the row and sends a fresh compare-and-swap from
+  it](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/correctness.md#row-write-settled), at most three
+  times. A generation's object is sent again, up to three more times, only after a `503` (`SlowDown` is the usual one) or a bare `429` ([why that is
+  safe](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/correctness.md#object-write-resent));
+  it carries a random id in its user metadata (`x-amz-meta-cbwid`), so a first send that landed is told from another
+  writer's object. A load whose row writes all go unanswered throws `TransientError` and keeps its object. The waits
+  before an object's re-sends use real timers: the store's injected clock does not reach them, and `S3Storage` takes no
+  clock, so a test that throttles one object write three times waits about 3.5 s at most (the publish's waits before a
+  fresh row write do use the store's clock). **A bare `429`** (AWS S3 does not send one; some S3-compatible services do)
+  **is a throttle, handled exactly as a `503` is**, and one that ends a call is a `TransientError`.
 - **Reads can be timed, and writes are not.** Nothing is timed unless you set `readTimeoutMs`. Set, a read that has
   not finished in that many ms throws `TransientError`, and the store runs it again; AWS's guidance is to retry a GET
   of under 512 KB after about 2 seconds. The timer covers the body as well, so a connection that sends its headers and
@@ -87,6 +111,22 @@ It builds its own client from your usual AWS credentials. Any other key is refus
   For the writes and listings, give your client a timeout of its own.
 
 The [production checklist](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/production.md) covers each item, with a sample IAM policy, and the ones every backend shares: a request timeout on your client for the requests the library does not time, backups of the data and the registry, and a schedule for your loads.
+- **The driver redacts the credentials it recognises from an error before it throws it.** The AWS SDK keeps the
+  response it got on the error (`$response`), and through its socket the request it sent, with the signed
+  `Authorization` header and `x-amz-security-token`. It also puts every field of the error body on the error, and S3
+  quotes the request back when it refuses it: its signature and canonical request (`SignatureDoesNotMatch`), or its
+  session token (`Token-0`, on `ExpiredToken` and `InvalidToken`). In the error the driver throws, the value under a
+  credential header or field name is replaced by `[redacted]` (`authorization`, `x-amz-security-token`,
+  `x-amz-s3session-token`, an SSE-C key, `SignatureProvided`, `CanonicalRequest`, `Token-0` and the like; the access
+  key id S3 names is kept), and so are the value beside a credential's name in a list or a record (a table row that
+  parses to `{ th, td }` with one name and one value, a `{ name, value }` pair, `ArgumentName` and `ArgumentValue`), a credential header line, a SigV4 signature
+  (`Signature=`, `X-Amz-Signature=`) and a session token or SSE-C key after its header name, anywhere in its text. A
+  credential in another form is not recognised, and a text too large for these patterns is replaced whole. An SDK
+  error that carries its transport or shows a credential, and every error it reaches (its `cause` chain, an
+  `AggregateError`'s `errors`), is thrown as a copy: a native error with the same prototype, name, message, stack,
+  `$metadata`, `$fault`, `Code` and request id, so classification is unchanged, without `$response`, the body the SDK
+  could not parse (`$responseBodyText`), any function or accessor property, or anything nested deeper than ten levels.
+  The live error is never changed, and an error with no transport and no credential in it is thrown as it is.
 
 ## Documentation
 

package/dist/backend.d.ts

@@ -1,26 +1,15 @@
-/**
- * `S3Storage` — the S3 backend as one object: the generations and the pointer, in one bucket, stated once.
- *
- * Replaces two constructor calls that each repeated `client`, `bucket` and `prefix`. Repeating them is how
- * they come apart: point the registry at one prefix and the objects at another and the store answers *empty*
- * rather than *misconfigured*, which is the hardest kind of wrong answer to debug. Here the location is written
- * once and shared, so the mismatch cannot be expressed.
- *
- * **It will build a client for you**, which is the common case — `new S3Storage({ bucket })` picks up the
- * ambient credential chain and region exactly as the SDK would. Pass `client` instead when you need a
- * credential chain the SDK cannot infer (SSO, an assumed role, a custom retry strategy); pass `endpoint` +
- * `pathStyle` + `credentials` for an S3-compatible store (MinIO, Ceph, R2). Both halves stay reachable as `.storage` and
- * `.registry` for anyone wiring something the facade does not cover. `maxObjectBytes` and `partBytes` size the
- * multipart upload, and `readTimeoutMs`, when set, bounds each read both halves make.
- */
 import { STORAGE_BACKEND } from '@cloudbitmaps/core/driver-kit';
-import type { IRegistryDriver, IStorageDriver, StorageBackend } from '@cloudbitmaps/core/driver-kit';
+import type { IRegistryDriver, IStorageDriver, LifecycleVerdict, StorageBackend } from '@cloudbitmaps/core/driver-kit';
 import { S3Client } from '@aws-sdk/client-s3';
 import type { IMetricsSink } from '@cloudbitmaps/core/driver-kit';
+/**
+ * What {@link S3Storage} takes: a `bucket`, and either a `client` or the five settings that build one (`region`,
+ * `endpoint`, `pathStyle`, `credentials`, `maxSockets`), never both.
+ */
 export interface S3StorageOptions {
     /** Target bucket (must already exist). */
     readonly bucket: string;
-    /** Optional key prefix under which everything lives — generations and the registry alike. */
+    /** Optional key prefix under which everything lives — generations and the registry alike. A string, else `ValidationError`. */
     readonly prefix?: string;
     /**
      * A constructed client. Supply one for a credential chain the SDK cannot infer; otherwise one is built. Its retry
@@ -28,14 +17,27 @@ export interface S3StorageOptions {
      * configured to do: a registry row once, and a generation's object again only after a throttle, under its own backoff.
      */
     readonly client?: S3Client;
-    /** Region for the client built when `client` is absent (refused beside `client`). Falls back to the SDK's own resolution. */
+    /**
+     * Region for the client built when `client` is absent (refused beside `client`): a non-empty string, else
+     * `ValidationError`. Falls back to the SDK's own resolution.
+     */
     readonly region?: string;
-    /** Endpoint for an S3-compatible store (MinIO, Ceph, R2). Refused beside `client`, which carries its own. */
+    /**
+     * Endpoint for an S3-compatible store (MinIO, Ceph, R2): an http or https URL, since an empty one would send
+     * requests to AWS, else `ValidationError`. Refused beside `client`, which carries its own.
+     */
     readonly endpoint?: string;
-    /** Path-style addressing, which most S3-compatible stores require. Refused beside `client`, which carries its own. */
+    /**
+     * Path-style addressing, which most S3-compatible stores require: a boolean, else `ValidationError`. Refused beside
+     * `client`, which carries its own.
+     */
     readonly pathStyle?: boolean;
     /**
-     * Static credentials, for the S3-compatible stores that issue them (MinIO, Ceph, R2).
+     * Static credentials, for the S3-compatible stores that issue them (MinIO, Ceph, R2): non-empty `accessKeyId` and
+     * `secretAccessKey` strings and an optional `sessionToken`, else `ValidationError`, which never prints a value. No other
+     * key is taken except `$source`, `credentialScope`, `accountId` and `expiration`, which are tolerated and not read:
+     * the SDK stamps `$source` on a credentials object it is given, so one used for two stores carries it, and a credential
+     * provider's result carries the others. A provider function is not taken here; put it on an `S3Client` given as `client`.
      *
      * On AWS itself, leave this unset — the SDK's own chain (instance role, SSO, environment, profile) is what
      * you want, and hard-coding keys to reach it would be a downgrade. It exists because the alternative for a
@@ -53,12 +55,13 @@ export interface S3StorageOptions {
      * two-operand `intersect`. Only this limit differs from the SDK's own client: its request handler, keep-alive,
      * timeouts and retry are the SDK's. If a future SDK changes its handler, the built client keeps the SDK's own
      * limit rather than failing to build. A positive safe integer. Release a built client's sockets with
-     * `store.client.destroy()`. Refused beside `client`, which carries its own request handler. A deployment
-     * that runs `eraseSubject`'s 256 reads at once needs `256`, or a lower `concurrency`.
+     * `store.client.destroy()`. Refused beside `client`, which carries its own request handler. An `eraseSubject`
+     * at its default `concurrency` of 8 holds up to 64 requests open, which the default of 128 covers; raise
+     * `maxSockets`, or lower `concurrency`, to fit a larger one.
      */
     readonly maxSockets?: number;
     /**
-     * Largest object the backend will write and advertise. Default = `partBytes × 10,000` (≈ 80 GiB at the default
+     * Largest object the backend will write and advertise. Default = `partBytes × 10,000` (≈ 78 GiB at the default
      * 8 MiB part) — the honest ceiling reachable within S3's 10,000-part limit. Set it higher and `partBytes`
      * auto-grows so 10,000 parts still cover it (raising peak write memory to ~one part); up to the 5 TiB S3 max, and a
      * larger cap, or a `partBytes` above S3's 5 GiB part limit, is refused with `ValidationError`.
@@ -115,6 +118,21 @@ export declare const S3_STORAGE_OPTION_KEYS: readonly ["bucket", "prefix", "clie
  * the agents it made.
  */
 export declare function limitSockets(client: S3Client, maxSockets: number): void;
+/**
+ * `S3Storage` — the S3 backend as one object: the generations and the pointer, in one bucket, stated once.
+ *
+ * Replaces two constructor calls that each repeated `client`, `bucket` and `prefix`. Repeating them is how
+ * they come apart: point the registry at one prefix and the objects at another and the store answers *empty*
+ * rather than *misconfigured*, which is the hardest kind of wrong answer to debug. Here the location is written
+ * once and shared, so the mismatch cannot be expressed.
+ *
+ * **It will build a client for you**, which is the common case — `new S3Storage({ bucket })` picks up the
+ * ambient credential chain and region exactly as the SDK would. Pass `client` instead when you need a
+ * credential chain the SDK cannot infer (SSO, an assumed role, a custom retry strategy); pass `endpoint` +
+ * `pathStyle` + `credentials` for an S3-compatible store (MinIO, Ceph, R2). Both halves stay reachable as `.storage`
+ * and `.registry` for anyone wiring something the facade does not cover. `maxObjectBytes` and `partBytes` size the
+ * multipart upload, and `readTimeoutMs`, when set, bounds each read both halves make (listings are not timed).
+ */
 export declare class S3Storage implements StorageBackend {
     /** Cross-bundle brand, stamped non-enumerably in the constructor so a spread cannot carry it. */
     readonly [STORAGE_BACKEND]: true;
@@ -123,10 +141,32 @@ export declare class S3Storage implements StorageBackend {
     /** The client both halves share — built here unless one was supplied. */
     readonly client: S3Client;
     private readonly sockets;
+    private readonly bucket;
+    /** The key prefix every object sits under: `''`, or `trimmed/`. */
+    private readonly keyPrefix;
+    private readonly readTimeoutMs;
+    /**
+     * Builds the S3 backend over one bucket, which `bucket` names. Pass `client`, or `region`, `endpoint`, `pathStyle`,
+     * `credentials` and `maxSockets` to have one built, not both. `prefix` puts every object under a key prefix. Throws
+     * `ValidationError` for an unknown option, no bucket, a `now` that is not a function, a client setting beside
+     * `client`, a `maxSockets` (with no `client`) that is not a positive integer, a `partBytes` or `maxObjectBytes` that
+     * is not a positive integer, a `conditionalDelete` that is not a boolean, a `region` or `endpoint` that is empty or not a string, a `pathStyle` that
+     * is not a boolean, `credentials` that are not complete, or an invalid `prefix` or `readTimeoutMs`.
+     */
     constructor(options: S3StorageOptions);
     /**
      * Called by a store given a metrics sink. After the first read, a client whose socket pool is smaller than twice the
      * default window (64) sends the sink one `advisory` event, once; see the production guide's socket sizing.
      */
     attachMetrics(sink: IMetricsSink): void;
+    /**
+     * Reads the bucket's lifecycle configuration (one `GetBucketLifecycleConfiguration` request, plus the SDK's own
+     * retries, bounded by `readTimeoutMs` when it is set) and reports a rule that would expire, or move to GLACIER,
+     * DEEP_ARCHIVE or a class it does not know, the current objects under this store's prefix, generations and registry
+     * rows alike. Needs `s3:GetLifecycleConfiguration`, which a load never uses (`store.preflight()`'s
+     * lifecycle step does): without it the
+     * answer is `unchecked`, naming the permission. A missing rule that aborts incomplete multipart uploads is advice in
+     * `detail`, not a problem. Never rejects; a `detail` carries no credential. Rule reading: see the production guide.
+     */
+    checkLifecycle(): Promise<LifecycleVerdict>;
 }