@cloudbitmaps/gcs 0.22.0 → 0.24.0

package/README.md

@@ -14,20 +14,14 @@ second service to run.
 pnpm add @cloudbitmaps/roaring @cloudbitmaps/gcs
 ```
 
-`roaring` needs its one install script to fetch its native binary, and newer clients run a dependency's script only if you allow it. Allow it before you install:
+Install the same version of every `@cloudbitmaps/*` package: they release together, and each pins the exact version of `core` it was built with.
 
-- **npm 12:** add `{ "allowScripts": { "roaring": true } }` to your `package.json`. Without it the install exits 0 and the package throws at `import`. npm 11 runs the script but warns until you allow it the same way.
-- **pnpm 11 and later:** put `allowBuilds: { roaring: true }` in a `pnpm-workspace.yaml` beside your `package.json`. Those versions no longer read the `package.json` `pnpm` key. Without it the install exits 1 with `ERR_PNPM_IGNORED_BUILDS`.
-- **pnpm 10:** add `{ "pnpm": { "onlyBuiltDependencies": ["roaring"] } }` to your `package.json`. Without it the install exits 0 and the package throws at `import`.
-- **pnpm 9:** nothing extra.
-
-[Details](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/getting-started.md#cannot-find-module-buildreleaseroaringnode-after-a-successful-install).
-
-`@google-cloud/storage` (`^7 || ^8`) is a real dependency of this package, so installing it is the whole step.
+`@google-cloud/storage` (`^8.1.0`) is a real dependency of this package, so installing it is the whole step. A `client` you pass must come from 8.1.0 or later: the driver's own tests pass on 8.1.0 and fail on every older release tried (8.0.1, 8.0.0, and 7.x from 7.0.0 to 7.22.0). On 8.0.1 the unit tests pass, but the integration test of a resumable upload's tail size against the emulator fails with a reset connection. On 8.0.0 and on 7.x the unit tests fail: a read the driver cuts off at `readTimeoutMs` leaves its connection open, a read whose first answer is cut off part-way is not retried to success (the call never settles), and a read the driver refuses does not release its request.
 `@cloudbitmaps/core`, the engine underneath, arrives with it and you never name it.
 
 ## Use
 
+<!-- typecheck -->
 ```ts
 import { CloudRoaring } from '@cloudbitmaps/roaring';
 import { GcsStorage } from '@cloudbitmaps/gcs';
@@ -56,12 +50,20 @@ It builds its own client from Application Default Credentials. Any other key is
 
 - **This backend is outside the validated envelope.** It passes the same conformance suites as S3, but the proven
   scale and the calibration were done on S3. You are an early user.
+- **Run `await store.preflight()` once before the first load.** It writes, tries to write again, reads, lists and deletes one probe
+  object, and reads the registry and the bucket's lifecycle rules (`storage.buckets.get` for the last), and answers
+  `{ ok, checks }`. It does not check a read after a delete or versioning, among others: [the full list](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/production.md#what-preflight-does-not-check). A first run is eight requests here.
 - **Grant the bucket's object permissions.** The library creates, reads, lists and deletes objects:
   `roles/storage.objectAdmin` covers all four.
 - **The write-once guarantee rides GCS preconditions** (`ifGenerationMatch`), so nothing else needs enabling.
 - **Never add a lifecycle rule that expires current objects or the `registry/` prefix.**
-- **The SDK does not retry downloads; the driver does.** In `@google-cloud/storage` 7.x and 8.x (checked on 7.22.0 and
-  8.1.0), a download the SDK retries after any status it retries (408, 429, 500, 502, 503 or 504) can crash the process
+- **Check the bucket's lifecycle rules with `await backend.checkLifecycle()`.** It reads the bucket's metadata with one
+  request (bounded by `readTimeoutMs`, when set) and answers `problem` for a `Delete` rule that applies to live objects and can match your prefix, `ok`, or
+  `unchecked`. It needs `storage.buckets.get`, which a load never uses, `store.preflight()`'s lifecycle step does, and `roles/storage.objectAdmin` does
+  not include: without it the answer is `unchecked`, naming the permission
+  ([check the lifecycle rules from code](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/production.md#check-the-lifecycle-rules-from-code)).
+- **The SDK does not retry downloads; the driver does.** In `@google-cloud/storage` 8.x (checked on 8.1.0) and 7.x (checked on
+  7.22.0, a release outside this driver's range), a download the SDK retries after any status it retries (408, 429, 500, 502, 503 or 504) can crash the process
   with `ERR_STREAM_UNABLE_TO_PIPE`, thrown outside any promise, even though the retried request succeeded. The client
   `GcsStorage` builds therefore sends each download once, and the driver runs a download again itself, up to three more
   times with backoff, after a connection fault (refused, reset, timed out, a DNS failure, a body cut off) or a 408, 429,
@@ -94,8 +96,9 @@ It builds its own client from Application Default Credentials. Any other key is
 - **A read that times out before its server answers leaves its connection open.** The SDK cannot cancel a request
   whose response has not begun, so that request holds its connection until the server answers or closes it, and until
   then keeps a Node process from exiting: one per driver read, up to four per call through the store's retry, so an
-  outage at N reads a second adds about 4N held connections each second. On 8.x a read cut off after its response
-  began closes its own; on 7.x (checked on 7.22.0) it stays open too. A 404 whose error body arrives after the
+  outage at N reads a second adds about 4N held connections each second. On every version this driver's range admits
+  (8.1.0 and later) a read cut off after its response began closes its own; on 8.0.0 and 7.x (checked on 7.22.0), both
+  outside the range, it stays open too. A 404 whose error body arrives after the
   deadline is reported as a `TransientError`, not `NotFoundError`.
 - **Downloads go out on Node's global agent, so one the driver cuts off or refuses resets nothing else.** When a
   download fails part-way, the SDK destroys the HTTP agent it went out on, and its own agent is one keep-alive pool the
@@ -105,15 +108,17 @@ It builds its own client from Application Default Credentials. Any other key is
   global agent closes a connection idle for 5 seconds (the SDK's own pool does not), so a read after a longer pause
   opens a new connection, with its TCP and TLS handshake, and downloads share that agent with any other `http` or
   `https` request in the process. Raising `https.globalAgent.options.timeout` keeps idle connections longer.
-- **A transient failure of a write throws `TransientError`, and the write may or may not have landed.** Re-run the
-  call, or check `store.generations(ref)`. The driver sends each registry write once; a load that gets no answer reads
-  the row and sends a fresh compare-and-swap from it, at most three times. A generation's object of up to
-  `simpleUploadThresholdBytes` is sent again, up to three more times, only after a `429` or `503`; every object carries
-  a random id in its metadata, so a first send that landed is told from another writer's object. A load whose row writes
-  all go unanswered throws `TransientError` and keeps its object. The waits before an object's re-sends use real
-  timers: the store's injected clock does not reach them, and `GcsStorage` takes no clock, so a test that throttles
-  one object write three times waits about 3.5 s at most (the publish's waits before a fresh row write do use the
-  store's clock).
+- **A transient failure of a write throws `TransientError`, and the write may or may not have landed.** Re-run the call,
+  or check `store.generations(ref)`. The driver [sends each registry write
+  once](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/correctness.md#sdk-retry); a load that gets no
+  answer [reads the row and sends a fresh compare-and-swap from
+  it](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/correctness.md#row-write-settled), at most three
+  times. A generation's object of up to `simpleUploadThresholdBytes` is sent again, up to three more times, only after a
+  `429` or `503`; every object carries a random id in its metadata, so a first send that landed is told from another
+  writer's object. A load whose row writes all go unanswered throws `TransientError` and keeps its object. The waits
+  before an object's re-sends use real timers: the store's injected clock does not reach them, and `GcsStorage` takes no
+  clock, so a test that throttles one object write three times waits about 3.5 s at most (the publish's waits before a
+  fresh row write do use the store's clock).
 
 The [production checklist](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/production.md) covers each item, and the ones every backend shares: backups of the data and the registry, and a schedule for your loads.
 - **The driver redacts the credentials it recognises from an error before it throws it.** The SDK keeps the response it

package/dist/backend.d.ts

@@ -1,5 +1,5 @@
 import { STORAGE_BACKEND } from '@cloudbitmaps/core/driver-kit';
-import type { IRegistryDriver, IStorageDriver, StorageBackend } from '@cloudbitmaps/core/driver-kit';
+import type { IRegistryDriver, IStorageDriver, LifecycleVerdict, StorageBackend } from '@cloudbitmaps/core/driver-kit';
 import { Storage as GcsClient } from '@google-cloud/storage';
 /**
  * What {@link GcsStorage} takes: a `bucket`, and either a `client` or the two settings that build one (`projectId`,
@@ -8,13 +8,13 @@ import { Storage as GcsClient } from '@google-cloud/storage';
 export interface GcsStorageOptions {
     /** Target bucket (must already exist). */
     readonly bucket: string;
-    /** Optional object-name prefix under which everything lives — generations and the registry alike. */
+    /** Optional object-name prefix under which everything lives — generations and the registry alike. A string, else `ValidationError`. */
     readonly prefix?: string;
     /**
      * A constructed `@google-cloud/storage` client. One is built from the ambient credentials when absent. Its retry
      * options apply to every request except the single-request conditional writes, which the driver sends without
      * that retry whatever they say (a registry row once, and a generation's object again only after a throttle, under
-     * its own backoff), and the downloads. In `@google-cloud/storage` 7.x and 8.x a download the SDK retries after a 408,
+     * its own backoff), and the downloads. In `@google-cloud/storage` 8.x, and in 7.x (which the driver's range does not admit), a download the SDK retries after a 408,
      * 429, 500, 502, 503 or 504 can crash the process with `ERR_STREAM_UNABLE_TO_PIPE`, so downloads go through a twin of
      * this client, built from its own class with the same credentials object, endpoint and settings and the SDK's retries
      * off over any its class set, and the driver retries them itself. A client whose twin would use other credentials or
@@ -25,9 +25,15 @@ export interface GcsStorageOptions {
      * built, other than an interceptor, is not carried to the twin.
      */
     readonly client?: GcsClient;
-    /** Project id for the client built when `client` is absent (refused beside `client`). Falls back to the SDK's own resolution. */
+    /**
+     * Project id for the client built when `client` is absent (refused beside `client`): a non-empty string, else
+     * `ValidationError`. Falls back to the SDK's own resolution.
+     */
     readonly projectId?: string;
-    /** Endpoint override — point it at fake-gcs-server locally. Refused beside `client`, which carries its own. */
+    /**
+     * Endpoint override — point it at fake-gcs-server locally: a non-empty string, since an empty one would send
+     * requests to the public endpoint, else `ValidationError`. Refused beside `client`, which carries its own.
+     */
     readonly apiEndpoint?: string;
     /** Largest object the backend will write and advertise (default = GCS's 5 TiB max). Must be a positive safe integer. */
     readonly maxObjectBytes?: number;
@@ -88,5 +94,25 @@ export declare class GcsStorage implements StorageBackend {
     readonly registry: IRegistryDriver;
     /** The client both halves share — built here unless one was supplied. */
     readonly client: GcsClient;
+    private readonly bucket;
+    /** The name prefix every object sits under: `''`, or `trimmed/`. */
+    private readonly keyPrefix;
+    private readonly readTimeoutMs;
+    /**
+     * Builds the GCS backend over one bucket, which `bucket` names. Pass `client`, or `projectId` and `apiEndpoint` to
+     * have one built, not both. `prefix` puts every object under a name prefix. Throws `ValidationError` for an unknown
+     * option, no bucket, a `now` that is not a function, a client setting beside `client`, a `maxObjectBytes` or
+     * `simpleUploadThresholdBytes` that is not a positive integer, a `conditionalDelete` that is not a boolean, an
+     * empty or non-string `apiEndpoint` or `projectId`, or an invalid `prefix` or `readTimeoutMs`.
+     */
     constructor(options: GcsStorageOptions);
+    /**
+     * Reads the bucket's metadata (one request, plus the SDK's own retries, bounded by `readTimeoutMs` when it is set) and
+     * reports a lifecycle rule with a `Delete` action (or one it does not know) that applies to live
+     * objects whose names can be this store's: generations and registry rows alike. Needs `storage.buckets.get`, which
+     * a load never uses (`store.preflight()`'s lifecycle step does) and `roles/storage.objectAdmin` does not include: without it the answer is
+     * `unchecked`, naming the permission. Never rejects; a `detail` carries no credential. Rule reading: see the
+     * production guide.
+     */
+    checkLifecycle(): Promise<LifecycleVerdict>;
 }