@cloudbitmaps/gcs 0.18.2 → 0.19.0

package/README.md

@@ -39,7 +39,7 @@ It builds its own client from Application Default Credentials. Any other key is
 |---|---|
 | `bucket` (required) | the bucket |
 | `prefix` | a key prefix for everything this store writes |
-| `client` | your own `Storage` client. Pass it as `client`; `storage` is refused, since in `CloudRoaring` that word means the backend. Build it with `retryOptions: { autoRetry: false }` (see below) |
+| `client` | your own `Storage` client. Pass it as `client`; `storage` is refused, since in `CloudRoaring` that word means the backend. Its downloads go through a twin of it with the SDK's retries off, and its single-request conditional writes are sent without them; a client no twin can be built from safely, such as a test stub, is refused (see below) |
 | `projectId`, `apiEndpoint` | build a client for you, such as one for fake-gcs-server; refused beside `client` |
 | `simpleUploadThresholdBytes`, `maxObjectBytes` | the size up to which an object is one simple request (default 8 MiB) and the largest object (default 5 TiB, GCS's maximum) |
 | `readTimeoutMs` | cut off a read that has run this long, in milliseconds (see below); `0`, the default, sets no timeout |
@@ -54,14 +54,25 @@ It builds its own client from Application Default Credentials. Any other key is
 - **The write-once guarantee rides GCS preconditions** (`ifGenerationMatch`), so nothing else needs enabling.
 - **Never add a lifecycle rule that expires current objects or the `registry/` prefix.**
 - **The SDK does not retry downloads; the driver does.** In `@google-cloud/storage` 7.x and 8.x (checked on 7.22.0 and
-  8.1.0), a download the SDK retries after any status it retries (408, 429, 500, 502, 503 or 504) can crash the process with
-  `ERR_STREAM_UNABLE_TO_PIPE`, thrown outside any promise, even though the retried request succeeded. The client
-  `GcsStorage` builds therefore sends each download once, and the driver runs a download again itself, up to three
-  more times with backoff, after a connection fault (refused, reset, timed out, a DNS failure, a body cut off) or a 408, 429, 500, 502, 503 or 504, and after nothing else (not a missing credentials file or a TLS failure). That holds for every
-  caller, including a store built with `retry: false`. The client's other requests (uploads, listings, metadata reads)
-  keep the SDK's retries. **A client you pass as `client` is used as it is, so build it with
-  `retryOptions: { autoRetry: false }`.** That also turns off the SDK's retries of listings, metadata reads and
-  resumable uploads on that client, which the library does not retry; the client `GcsStorage` builds keeps them.
+  8.1.0), a download the SDK retries after any status it retries (408, 429, 500, 502, 503 or 504) can crash the process
+  with `ERR_STREAM_UNABLE_TO_PIPE`, thrown outside any promise, even though the retried request succeeded. The client
+  `GcsStorage` builds therefore sends each download once, and the driver runs a download again itself, up to three more
+  times with backoff, after a connection fault (refused, reset, timed out, a DNS failure, a body cut off) or a 408, 429,
+  500, 502, 503 or 504, and after nothing else (not a missing credentials file or a TLS failure). That holds for every
+  caller, including a store built with `retry: false`. The client's listings, deletes, resumable uploads and the
+  metadata read that settles a write keep its retry settings (under which the SDK sends a delete with no precondition
+  once); a single-request conditional write (a registry row, a small generation) is sent without them. **A client you
+  pass as `client` keeps its own retries for those requests**: its downloads go through a twin of it, built from its own
+  class with the same credentials object, endpoint, project, user agent, timeout, checksum generator and interceptors
+  and the SDK's retries off. A `Storage` client is always used through its twin, which must have taken the settings that
+  decide where and how a download goes: the same credentials object, URL and endpoint, and retry settings of its own,
+  which are then turned off over any its class set. The backend refuses, when it is built, a client whose twin would not
+  (a subclass that builds from options of its own), one whose class overrides `bucket` (as a test double built on
+  `Storage` does), and one whose `bucket` is replaced on the instance (as by a test stub). Stub `Storage.prototype` (or
+  `Bucket.prototype`, `File.prototype`) instead, which reaches the twin; a stub on the client's instance does not. A
+  test double that is not a `Storage` client is used as it is when it has `retryOptions: { autoRetry: false }`, and
+  refused without it; with the SDK's module mocked, pass the mock as `client`. What is set on a client after the backend
+  was built, other than an interceptor, is not carried to the twin.
 - **A client `timeout` does not bound a download on `@google-cloud/storage` 8.x**, so a read whose server stalls waits
   for it. Measured against a local server that never answers: still pending after 12 s with `timeout: 2000`.
   **`readTimeoutMs` does, and it is off unless you set it.** It bounds each read as a whole: a generation's tail (with

package/dist/backend.d.ts

@@ -19,11 +19,16 @@ export interface GcsStorageOptions {
     /**
      * A constructed `@google-cloud/storage` client. One is built from the ambient credentials when absent. Its retry
      * options apply to every request except the single-request conditional writes, which the driver sends without
-     * that retry whatever they say: a registry row once, and a generation's object again only after a throttle, under
-     * its own backoff. In `@google-cloud/storage` 7.x and 8.x a download the SDK retries after a 408, 429, 500, 502, 503 or 504 can crash the process
-     * with `ERR_STREAM_UNABLE_TO_PIPE`, so build it with `retryOptions: { autoRetry: false }`; the driver retries
-     * downloads itself. That also turns off the SDK's retries of listings, metadata reads and resumable uploads on
-     * this client, which the store does not retry. The client built here needs none of this: only its downloads are sent once.
+     * that retry whatever they say (a registry row once, and a generation's object again only after a throttle, under
+     * its own backoff), and the downloads. In `@google-cloud/storage` 7.x and 8.x a download the SDK retries after a 408,
+     * 429, 500, 502, 503 or 504 can crash the process with `ERR_STREAM_UNABLE_TO_PIPE`, so downloads go through a twin of
+     * this client, built from its own class with the same credentials object, endpoint and settings and the SDK's retries
+     * off over any its class set, and the driver retries them itself. A client whose twin would use other credentials or
+     * another endpoint (a subclass that builds from options of its own), whose class overrides `bucket` (a test double
+     * built on `Storage`), or whose `bucket` is replaced on the instance (a test stub; stub `Storage.prototype` instead)
+     * is refused at construction. A test double that is not a `Storage` client is used as it is when it has
+     * `retryOptions: { autoRetry: false }`, and refused without it. What is set on the client after the backend was
+     * built, other than an interceptor, is not carried to the twin.
      */
     readonly client?: GcsClient;
     /** Project id for the client built when `client` is absent (refused beside `client`). Falls back to the SDK's own resolution. */

package/dist/bucket-presence.d.ts

@@ -0,0 +1,37 @@
+/**
+ * Whether the bucket a driver names exists, asked when a request first answers 404.
+ *
+ * GCS answers a missing bucket and a missing object with the same status and reason (`404 notFound`), and a
+ * GCS-compatible server may not even word them apart. Read as a missing object, a misnamed or deleted bucket would
+ * answer every read as an empty segment. An object listing tells them apart: it names no object, so its 404 can only
+ * be the bucket's.
+ *
+ * ```
+ *   a request answers 404 ──▶ confirm() ──▶ bucket seen before? ── yes ──▶ the 404 is an object's
+ *                                              │ no
+ *                                              ▼
+ *                              list one object in the bucket
+ *                               ├─ answered ──────────▶ remember the bucket; the 404 is an object's
+ *                               ├─ 404 ───────────────▶ reject: the bucket does not exist
+ *                               ├─ transient ─────────▶ reject: TransientError (asked again next time)
+ *                               ├─ 403 ───────────────▶ remember it as unknowable; the 404 is an object's
+ *                               └─ anything else ─────▶ the 404 is an object's (asked again next time)
+ * ```
+ *
+ * A process pays one listing at most while its bucket exists, and concurrent 404s share it. An identity without
+ * `storage.objects.list` is answered `403`, which says nothing about the bucket, so its 404s read as they always did.
+ * Any other failure (a credential that could not be fetched, a 401) says nothing either, but may not last, so it is
+ * not remembered. Once seen, the bucket is not asked about again: one deleted later in the process's life reads as an
+ * empty store, as it did before this check, while every write to it fails.
+ */
+import type { Storage } from '@google-cloud/storage';
+export declare class BucketPresence {
+    private readonly storage;
+    private readonly bucket;
+    private settled;
+    private asking;
+    constructor(storage: Storage, bucket: string);
+    /** Resolve when a 404 can be read as an object's; reject when the bucket does not exist or the listing failed in transit. */
+    confirm(): Promise<void>;
+    private ask;
+}

package/dist/crc32c.d.ts

@@ -0,0 +1,5 @@
+/**
+ * CRC32C (Castagnoli) of some bytes, in the form GCS takes in an object's `crc32c` metadata: the 4-byte big-endian
+ * value, base64-encoded. Computed here rather than through the SDK so it is the same on every major the driver accepts.
+ */
+export declare function crc32cBase64(bytes: Uint8Array): string;

package/dist/gcs-errors.d.ts

@@ -18,8 +18,15 @@ export declare function isPreconditionFailed(err: unknown): boolean;
  * an upload sent again after one must still tell a landed first send apart.
  */
 export declare function isThrottle(err: unknown): boolean;
-/** The object / generation does not exist (GCS returns 404). */
+/** The object / generation does not exist (GCS returns 404), in a bucket that does. */
 export declare function isNotFound(err: unknown): boolean;
+/**
+ * The bucket itself does not exist: a wiring fault, never an absent object. Read as one, a misnamed or deleted bucket
+ * would answer every read as an empty segment. GCS gives both the same status and reason (`404 notFound`), so only its
+ * message tells them apart: on a download the message is the response body as it came, JSON or text, and elsewhere
+ * the message and each `errors[].message` are GCS's own. A wording this does not match reads as an absent object.
+ */
+export declare function isMissingBucket(err: unknown): boolean;
 /** A range request started past EOF (HTTP 416 Requested Range Not Satisfiable). */
 export declare function isInvalidRange(err: unknown): boolean;
 /** A download that failed in transit rather than with an HTTP status: one of {@link TRANSPORT_CODES}. */