@cloudbitmaps/gcs 0.11.1 → 0.12.0

package/README.md

@@ -39,9 +39,11 @@ It builds its own client from Application Default Credentials. Any other key is
 |---|---|
 | `bucket` (required) | the bucket |
 | `prefix` | a key prefix for everything this store writes |
-| `client` | your own `Storage` client. Pass it as `client`; `storage` is refused, since in `CloudRoaring` that word means the backend |
+| `client` | your own `Storage` client. Pass it as `client`; `storage` is refused, since in `CloudRoaring` that word means the backend. Build it with `retryOptions: { autoRetry: false }` (see below) |
 | `projectId`, `apiEndpoint` | build a client for you, such as one for fake-gcs-server; refused beside `client` |
 | `simpleUploadThresholdBytes`, `maxObjectBytes` | the size up to which an object is one simple request (default 8 MiB) and the largest object (default 5 TiB, GCS's maximum) |
+| `readTimeoutMs` | cut off a read that has run this long, in milliseconds (see below); `0`, the default, sets no timeout |
+| `conditionalDelete` | whether the registry removes a deleted row with an object delete under `ifGenerationMatch`, rather than leaving a tombstone every full listing reads. Off by default, on the public endpoint too: whether real GCS applies `ifGenerationMatch` to a delete has not been verified by a run against the service, and fake-gcs-server ignores it, so CI cannot show it. Set `true` to remove rows for good. Needs delete permission on the registry prefix |
 
 ## Before production
 
@@ -51,10 +53,51 @@ It builds its own client from Application Default Credentials. Any other key is
   `roles/storage.objectAdmin` covers all four.
 - **The write-once guarantee rides GCS preconditions** (`ifGenerationMatch`), so nothing else needs enabling.
 - **Never add a lifecycle rule that expires current objects or the `registry/` prefix.**
+- **The SDK does not retry downloads; the driver does.** In `@google-cloud/storage` 7.x and 8.x (checked on 7.22.0 and
+  8.1.0), a download the SDK retries after any status it retries (408, 429, 500, 502, 503 or 504) can crash the process with
+  `ERR_STREAM_UNABLE_TO_PIPE`, thrown outside any promise, even though the retried request succeeded. The client
+  `GcsStorage` builds therefore sends each download once, and the driver runs a download again itself, up to three
+  more times with backoff, after a connection fault (refused, reset, timed out, a DNS failure, a body cut off) or a 408, 429, 500, 502, 503 or 504, and after nothing else (not a missing credentials file or a TLS failure). That holds for every
+  caller, including a store built with `retry: false`. The client's other requests (uploads, listings, metadata reads)
+  keep the SDK's retries. **A client you pass as `client` is used as it is, so build it with
+  `retryOptions: { autoRetry: false }`.** That also turns off the SDK's retries of listings, metadata reads and
+  resumable uploads on that client, which the library does not retry; the client `GcsStorage` builds keeps them.
+- **A client `timeout` does not bound a download on `@google-cloud/storage` 8.x**, so a read whose server stalls waits
+  for it. Measured against a local server that never answers: still pending after 12 s with `timeout: 2000`.
+  **`readTimeoutMs` does, and it is off unless you set it.** It bounds each read as a whole: a generation's tail (with
+  the metadata read it falls back on for an empty object), a range of it, or a registry row. One deadline covers every
+  attempt the driver makes and the backoff between them, from the call into the driver (a credential fetch and any
+  wait for a socket count) to the end of the body. It counts time the process spends busy too: Node runs a due timer
+  before it reads a socket, so a synchronous stretch longer than the timeout fails the reads in flight even when their
+  responses have arrived. When it passes, the read throws `TransientError` naming the read and the timeout, and no
+  further attempt starts; the store's read retry runs it again, so with `readTimeoutMs: 2_000` a read that stalls on
+  every attempt fails after about 8.35 s (4 × 2 s plus up to 0.35 s of backoff; measured: 8.1 to 8.2 s). Uploads,
+  deletes and listings are not timed.
+- **A read that times out before its server answers leaves its connection open.** The SDK cannot cancel a request
+  whose response has not begun, so that request holds its connection until the server answers or closes it, and until
+  then keeps a Node process from exiting: one per driver read, up to four per call through the store's retry, so an
+  outage at N reads a second adds about 4N held connections each second. On 8.x a read cut off after its response
+  began closes its own; on 7.x (checked on 7.22.0) it stays open too. A 404 whose error body arrives after the
+  deadline is reported as a `TransientError`, not `NotFoundError`.
+- **Downloads go out on Node's global agent, so one the driver cuts off or refuses resets nothing else.** When a
+  download fails part-way, the SDK destroys the HTTP agent it went out on, and its own agent is one keep-alive pool the
+  whole process shares, so every other request in flight would be reset with it, uploads included. The SDK never
+  destroys the global agent. Measured on 8.1.0 with no timeout: a body cut off part-way reset a 64-byte upload and a
+  registry write on a second backend on the SDK's agent, and on the global agent all of them completed. The cost: the
+  global agent closes a connection idle for 5 seconds (the SDK's own pool does not), so a read after a longer pause
+  opens a new connection, with its TCP and TLS handshake, and downloads share that agent with any other `http` or
+  `https` request in the process. Raising `https.globalAgent.options.timeout` keeps idle connections longer.
 - **A transient failure of a write throws `TransientError`, and the write may or may not have landed.** Re-run the
-  call, or check `store.generations(ref)`.
-
-The [production checklist](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/production.md) covers each item, and the ones every backend shares: a request timeout on your client, backups of the data and the registry, and a schedule for your loads.
+  call, or check `store.generations(ref)`. The driver sends each registry write once; a load that gets no answer reads
+  the row and sends a fresh compare-and-swap from it, at most three times. A generation's object of up to
+  `simpleUploadThresholdBytes` is sent again, up to three more times, only after a `429` or `503`; every object carries
+  a random id in its metadata, so a first send that landed is told from another writer's object. A load whose row writes
+  all go unanswered throws `TransientError` and keeps its object. The waits before an object's re-sends use real
+  timers: the store's injected clock does not reach them, and `GcsStorage` takes no clock, so a test that throttles
+  one object write three times waits about 3.5 s at most (the publish's waits before a fresh row write do use the
+  store's clock).
+
+The [production checklist](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/production.md) covers each item, and the ones every backend shares: backups of the data and the registry, and a schedule for your loads.
 
 ## Documentation
 

package/dist/backend.d.ts

@@ -18,8 +18,12 @@ export interface GcsStorageOptions {
     readonly prefix?: string;
     /**
      * A constructed `@google-cloud/storage` client. One is built from the ambient credentials when absent. Its retry
-     * options apply to every request except the single-request conditional writes, which are sent once whatever they
-     * say.
+     * options apply to every request except the single-request conditional writes, which the driver sends without
+     * that retry whatever they say: a registry row once, and a generation's object again only after a throttle, under
+     * its own backoff. In `@google-cloud/storage` 7.x and 8.x a download the SDK retries after a 408, 429, 500, 502, 503 or 504 can crash the process
+     * with `ERR_STREAM_UNABLE_TO_PIPE`, so build it with `retryOptions: { autoRetry: false }`; the driver retries
+     * downloads itself. That also turns off the SDK's retries of listings, metadata reads and resumable uploads on
+     * this client, which the store does not retry. The client built here needs none of this: only its downloads are sent once.
      */
     readonly client?: GcsClient;
     /** Project id for the client built when `client` is absent (refused beside `client`). Falls back to the SDK's own resolution. */
@@ -33,13 +37,38 @@ export interface GcsStorageOptions {
     readonly simpleUploadThresholdBytes?: number;
     /** Injected clock for the registry's `createdAt`/`updatedAt`; defaults to `Date.now`. */
     readonly now?: () => number;
+    /**
+     * Cut off a read that has run this long, in milliseconds. `0`, the default, sets no timeout. A client's own `timeout`
+     * does not bound a download on `@google-cloud/storage` 8.x (the SDK hands it to an HTTP client that has no such
+     * option), so this is what does.
+     *
+     * It bounds each read as a whole — a generation's tail (with the metadata read it falls back on for an empty
+     * object), a range of it, a registry row — with one deadline across every attempt the driver makes and the backoff
+     * between them. The clock starts at the call into the driver, so a credential fetch and any wait for a socket count
+     * (Node's agents set no socket limit unless your process sets one), and runs until the whole body has arrived. It
+     * counts time the process spends busy too: Node runs a due timer before it reads a socket, so a synchronous stretch
+     * longer than the timeout fails the reads in flight even when their responses have arrived. When it passes, the read
+     * throws `TransientError` naming the read and the timeout and no further attempt starts, for the store's read retry
+     * to run again: about 8.35 s in all at `2_000` with the default retry policy. Uploads, deletes, listings and the
+     * conditional writes are not timed. The SDK cannot cancel a request whose response has not begun, so a read timed
+     * out before its server answers leaves that connection open until the server answers or closes it: one per read, up
+     * to four per call through the store's retry. A non-negative safe integer no larger than 2,147,483,647.
+     */
+    readonly readTimeoutMs?: number;
+    /**
+     * Whether the registry removes a deleted row for good, by an object delete sent with `ifGenerationMatch`, rather
+     * than leaving a tombstone a full listing reads forever. Defaults to `false`, on the public endpoint too: whether real
+     * GCS applies `ifGenerationMatch` to a delete has not been verified by a run against the service, and fake-gcs-server
+     * accepts the precondition and ignores it, so CI cannot show it. Set `true` to remove rows for good.
+     */
+    readonly conditionalDelete?: boolean;
 }
 /**
  * The keys `new GcsStorage(options)` takes. Any other is refused by name rather than ignored: an ignored client key
  * — `storage`, as the lower-level storage driver calls it — falls back to ambient credentials and the **public** endpoint, and
  * for a client pointed at an emulator that is production traffic from a wiring typo.
  */
-export declare const GCS_STORAGE_OPTION_KEYS: readonly ["bucket", "prefix", "client", "projectId", "apiEndpoint", "maxObjectBytes", "simpleUploadThresholdBytes", "now"];
+export declare const GCS_STORAGE_OPTION_KEYS: readonly ["bucket", "prefix", "client", "projectId", "apiEndpoint", "maxObjectBytes", "simpleUploadThresholdBytes", "now", "readTimeoutMs", "conditionalDelete"];
 export declare class GcsStorage implements StorageBackend {
     /** Cross-bundle brand, stamped non-enumerably in the constructor so a spread cannot carry it. */
     readonly [STORAGE_BACKEND]: true;

package/dist/download-retry.d.ts

@@ -0,0 +1,8 @@
+import { type Deadline } from './read-timeout.js';
+/** Under a deadline, a retry is not sent with less than this left of it. */
+export declare const MIN_RETRY_MS = 10;
+/**
+ * Call `download`, and again after each retryable fault, up to {@link RETRIES} more times, within `deadline` when there
+ * is one; rethrow the last error, or the deadline's once it has passed.
+ */
+export declare function retryDownload<T>(download: () => Promise<T>, deadline?: Deadline): Promise<T>;

package/dist/gcs-errors.d.ts

@@ -12,12 +12,27 @@
  * returns **412 Precondition Failed**. Maps to `WriteConflictError` (caller OCC), never a blind retry.
  */
 export declare function isPreconditionFailed(err: unknown): boolean;
+/**
+ * GCS asked the client to slow down: `429` (`rateLimitExceeded`) or `503` (`backendError`), which the SDK raises as an
+ * `ApiError` with the status as a numeric `code`. GCS does not document that a throttled request was not applied, so
+ * an upload sent again after one must still tell a landed first send apart.
+ */
+export declare function isThrottle(err: unknown): boolean;
 /** The object / generation does not exist (GCS returns 404). */
 export declare function isNotFound(err: unknown): boolean;
 /** A range request started past EOF (HTTP 416 Requested Range Not Satisfiable). */
 export declare function isInvalidRange(err: unknown): boolean;
+/** A download that failed in transit rather than with an HTTP status: one of {@link TRANSPORT_CODES}. */
+export declare function isTransportFault(err: unknown): boolean;
+/**
+ * Whether the driver retries a failed download, which it does in the SDK's place: the statuses the SDK's predicate
+ * retries (408, 429, 500, 502, 503 and 504, as a number or a string `code`), the connection faults it names in any
+ * `errors[].reason`, and every {@link isTransportFault}, which the SDK retried before any response came. A 404, 412,
+ * 416 or 403, a 501 or 505, a credentials or TLS failure, and every other answer reach the caller on the first attempt.
+ */
+export declare function isDownloadRetryable(err: unknown): boolean;
 /**
- * A transient GCS fault that is safe to retry: throttling (429), any 5xx, or a dropped/timed-out socket.
+ * A transient GCS fault that is safe to retry: throttling (429), a request timeout (408), any 5xx, or a dropped/timed-out socket.
  * Excludes the deterministic, caller-meaningful outcomes (412/404/416) — those must never be reclassified as
  * a blind transient (a retried doomed conditional write would just fail again, and mask an OCC conflict).
  */