@cloudbitmaps/s3 0.11.1 → 0.12.0

package/README.md

@@ -16,7 +16,7 @@ pnpm add @cloudbitmaps/roaring @cloudbitmaps/s3
 
 On npm 12 and pnpm 10 and later, allow `roaring`'s one install script first, or the install exits 0 and the package throws at `import`. Put this in your `package.json`: `{ "allowScripts": { "roaring": true }, "pnpm": { "onlyBuiltDependencies": ["roaring"] } }`. npm 11 runs the script but warns until you allow it the same way; pnpm 9 needs nothing extra. [Details](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/getting-started.md#cannot-find-module-buildreleaseroaringnode-after-a-successful-install).
 
-`@aws-sdk/client-s3` (`>=3.645.0 <4`) is a real dependency of this package, so installing it is the whole step.
+`@aws-sdk/client-s3` (`>=3.700.0 <4`) is a real dependency of this package, so installing it is the whole step.
 `@cloudbitmaps/core`, the engine underneath, arrives with it and you never name it.
 
 ## Use
@@ -42,26 +42,46 @@ It builds its own client from your usual AWS credentials. Any other key is refus
 | `client` | your own `S3Client`; it carries its own region, endpoint and credentials |
 | `region`, `endpoint`, `pathStyle`, `credentials` | build a client for you, such as one for MinIO; refused beside `client` |
 | `partBytes`, `maxObjectBytes` | multipart sizing: part size (default 8 MiB; the upload buffers one part at a time) and the largest object (about 80 GiB by default, up to S3's 5 TiB) |
+| `readTimeoutMs` | how long one read (a `GetObject` or `HeadObject`, its body included) may take before it throws `TransientError` and the store retries it; `0`, the default, sets no timeout. Writes are not timed |
+| `conditionalDelete` | whether the registry removes a deleted row with a `DeleteObject` under `If-Match` rather than leaving a tombstone. On by default when the host the client resolves is an AWS S3 host, however its endpoint was set (`endpoint`, `AWS_ENDPOINT_URL_S3`, `AWS_ENDPOINT_URL`, the shared config file); off for any other host: set it for an S3-compatible store only once you know it applies the header (MinIO ignores it) |
 
 ## Before production
 
 - **The bucket must honor conditional writes**: `If-None-Match: *` for a write-once generation and `If-Match` for the
   pointer. AWS S3 does, and the test suite runs against MinIO. Another S3-compatible service that ignores the headers
   turns write-once into overwrite, so check yours before you depend on it.
-- **Use `@aws-sdk/client-s3` 3.645.0 or later** if you pin the SDK or pass your own `client`. This floor is a
-  correctness floor, not a preference. The SDK models the conditional write this library's write-once guarantee is
-  built on from 3.641.0: measured against MinIO, 3.640.0 silently overwrites an existing object, which loses a
-  published generation without an error, and 3.641.0 rejects correctly. The floor sits a small margin above that.
-  This package's own range never resolves below it.
+- **Use `@aws-sdk/client-s3` 3.700.0 or later** if you pin the SDK or pass your own `client`. This floor is a
+  correctness floor, not a preference. An SDK sends only the conditional headers it models, and drops one it does
+  not, without an error. `If-None-Match`, which write-once is built on, is modelled from 3.641.0: measured against
+  MinIO, 3.640.0 silently overwrites an existing object, which loses a published generation. `If-Match` on
+  `PutObject`, which the registry's compare-and-swap is built on, is modelled from 3.700.0: 3.699.0's serializer
+  omits it, so a fenced row write goes out unconditionally and can land over a concurrent writer's. This package's
+  own range never resolves below 3.700.0. The registry also checks, before its first request, that the client it is given sends `If-Match` on
+  `PutObject` and `DeleteObject`, and refuses a write it would send without its precondition.
 - **Grant `s3:GetObject`, `s3:PutObject`, `s3:DeleteObject`, `s3:AbortMultipartUpload` and `s3:ListBucket`.** Without
   `s3:ListBucket`, S3 answers a missing key with `403` instead of `404`.
 - **Add a lifecycle rule that aborts incomplete multipart uploads**, and never one that expires current objects or the
   `registry/` prefix.
 - **A transient failure of a write throws `TransientError`, and the write may or may not have landed.** Re-run the
   call, or check `store.generations(ref)`. The SDK's own retry is off for conditional writes, so a write that landed
-  is not reported as a conflict.
-
-The [production checklist](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/production.md) covers each item, with a sample IAM policy, and the ones every backend shares: a request timeout on your client, backups of the data and the registry, and a schedule for your loads.
+  is not reported as a conflict. The driver sends each registry write once; a load that gets no answer reads the row and
+  sends a fresh compare-and-swap from it, at most three times. A generation's object is sent again, up to three more
+  times, only after a `503` (`SlowDown` is the usual one); it carries a random id in its user metadata
+  (`x-amz-meta-cbwid`), so a first send that landed is told from another writer's object. A load whose row writes all go
+  unanswered throws `TransientError` and keeps its object. The waits before an object's re-sends use real timers: the
+  store's injected clock does not reach them, and `S3Storage` takes no clock, so a test that throttles one object
+  write three times waits about 3.5 s at most (the publish's waits before a fresh row write do use the store's
+  clock). **A bare `429`** (AWS S3 does not send one; some
+  S3-compatible services do) **is not retried and is not classified transient**: it surfaces as the SDK's own error, so a
+  layer of yours that keys on `TransientError` will not retry it.
+- **Reads can be timed, and writes are not.** Nothing is timed unless you set `readTimeoutMs`. Set, a read that has
+  not finished in that many ms throws `TransientError`, and the store runs it again; AWS's guidance is to retry a GET
+  of under 512 KB after about 2 seconds. The timer covers the body as well, so a connection that sends its headers and
+  then stalls is cut off too. It starts when the read is handed to the SDK, so a burst of reads larger than the
+  client's socket pool (50 by default) can time out while queued: set it above that queueing, or raise `maxSockets`.
+  For the writes and listings, give your client a timeout of its own.
+
+The [production checklist](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/production.md) covers each item, with a sample IAM policy, and the ones every backend shares: a request timeout on your client for the requests the library does not time, backups of the data and the registry, and a schedule for your loads.
 
 ## Documentation
 

package/dist/backend.d.ts

@@ -11,7 +11,7 @@
  * credential chain the SDK cannot infer (SSO, an assumed role, a custom retry strategy); pass `endpoint` +
  * `pathStyle` + `credentials` for an S3-compatible store (MinIO, Ceph, R2). Both halves stay reachable as `.storage` and
  * `.registry` for anyone wiring something the facade does not cover. `maxObjectBytes` and `partBytes` size the
- * multipart upload.
+ * multipart upload, and `readTimeoutMs`, when set, bounds each read both halves make.
  */
 import { STORAGE_BACKEND } from '@cloudbitmaps/core/driver-kit';
 import type { IRegistryDriver, IStorageDriver, StorageBackend } from '@cloudbitmaps/core/driver-kit';
@@ -23,7 +23,8 @@ export interface S3StorageOptions {
     readonly prefix?: string;
     /**
      * A constructed client. Supply one for a credential chain the SDK cannot infer; otherwise one is built. Its retry
-     * applies to every request except the conditional writes, which are sent once whatever it is configured to do.
+     * applies to every request except the conditional writes, which the driver sends with that retry off whatever it is
+     * configured to do: a registry row once, and a generation's object again only after a throttle, under its own backoff.
      */
     readonly client?: S3Client;
     /** Region for the client built when `client` is absent (refused beside `client`). Falls back to the SDK's own resolution. */
@@ -55,15 +56,40 @@ export interface S3StorageOptions {
     /** Multipart part size in bytes (default 8 MiB; a smaller value is raised to the S3 5 MiB minimum). Must be a
      * positive safe integer. Tunes peak write memory. */
     readonly partBytes?: number;
+    /**
+     * How long one read may take before it is abandoned, in ms. `0`, the default, sets no timeout. When set, it bounds
+     * each `GetObject` and `HeadObject` either half sends, the response body included, so a connection that stops
+     * answering part-way through a body is cut off too. A read that runs out of time throws `TransientError`, which the
+     * store's read retry runs again. AWS's S3 guidance is to retry a GET of under 512 KB that has not answered in about
+     * 2 seconds. Must be a non-negative safe integer no larger than 2,147,483,647.
+     *
+     * The clock starts when the read is handed to the SDK, so it also counts the time the read waits for one of the
+     * client's sockets (50 by default) and the time spent fetching credentials, and under `retryMode: 'adaptive'` the
+     * SDK's rate-limiter wait. A burst of concurrent reads larger than the socket pool can therefore time out with
+     * nothing slow on the wire: size the timeout above the worst queueing your concurrency implies, or raise the client's
+     * `maxSockets`. On a client built with `cacheMiddleware: true`, a timed read resolves its middleware each time.
+     *
+     * Writes and listings are never timed: a write that hangs needs a timeout on the client (its `requestHandler`). The
+     * timeout is applied per request, so a `client` you pass gets it without being changed.
+     */
+    readonly readTimeoutMs?: number;
     /** Injected clock for the registry's `createdAt`/`updatedAt`; defaults to `Date.now`. */
     readonly now?: () => number;
+    /**
+     * Whether the registry removes a deleted row for good, by a `DeleteObject` sent with `If-Match`, rather than leaving
+     * a tombstone a full listing reads forever. Defaults to `true` when the host the client resolves is an AWS S3 host,
+     * whichever way its endpoint was set (`endpoint`, `AWS_ENDPOINT_URL_S3`, `AWS_ENDPOINT_URL`, the shared config file),
+     * and to `false` for any other host: set it for an S3-compatible store only once you know the store applies `If-Match`
+     * on a delete. MinIO, for one, ignores it. It is never `true` for an SDK that does not send the header.
+     */
+    readonly conditionalDelete?: boolean;
 }
 /**
  * The keys `new S3Storage(options)` takes. Any other is refused by name rather than ignored: an ignored client or
  * endpoint key builds a client from ambient credentials against the **public** endpoint, and for a store pointed at
  * MinIO that is production traffic from a wiring typo.
  */
-export declare const S3_STORAGE_OPTION_KEYS: readonly ["bucket", "prefix", "client", "region", "endpoint", "pathStyle", "credentials", "maxObjectBytes", "partBytes", "now"];
+export declare const S3_STORAGE_OPTION_KEYS: readonly ["bucket", "prefix", "client", "region", "endpoint", "pathStyle", "credentials", "maxObjectBytes", "partBytes", "readTimeoutMs", "now", "conditionalDelete"];
 export declare class S3Storage implements StorageBackend {
     /** Cross-bundle brand, stamped non-enumerably in the constructor so a spread cannot carry it. */
     readonly [STORAGE_BACKEND]: true;

package/dist/client-probe.d.ts

@@ -0,0 +1,26 @@
+import type { S3Client } from '@aws-sdk/client-s3';
+/** What a client does with the requests the registry sends. */
+export interface ClientFacts {
+    /** The host a request to the bucket is addressed to, as the client resolves it now. */
+    readonly host: string;
+    /** Whether a `DeleteObject` given `IfMatch` goes out with an `If-Match` header. */
+    readonly sendsDeleteIfMatch: boolean;
+    /** Whether a `PutObject` given `IfMatch` goes out with an `If-Match` header: the registry's compare-and-swap. */
+    readonly sendsPutIfMatch: boolean;
+    /** Whether a `PutObject` given `IfNoneMatch` goes out with an `If-None-Match` header: the registry's create. */
+    readonly sendsPutIfNoneMatch: boolean;
+}
+/** A key no registry row or generation can hold; the probe's requests are never sent, so it is never used. */
+export declare const PROBE_KEY = "cloudbitmaps-probe";
+/**
+ * What `client` does with a conditional `DeleteObject` and `PutObject` to `bucket`, or `undefined` when it cannot be
+ * found out. Never throws, never sends a request, and never runs anything the caller added to `client`.
+ */
+export declare function probeClient(client: S3Client, bucket: string): Promise<ClientFacts | undefined>;
+/**
+ * Whether `hostname` is an AWS S3 host: under an AWS domain and naming S3 in one of its labels (`s3`, `s3-fips`,
+ * `s3-accesspoint`, `s3express-…`), so the FIPS, dual-stack, access-point and VPC interface forms all count, and another
+ * AWS service's host does not. A host of any other domain is not, whatever it speaks: a store behind one has to be
+ * vouched for by the caller.
+ */
+export declare function isAwsS3Host(hostname: string): boolean;