@cloudbitmaps/gcs 0.21.0 → 0.23.0

package/README.md

@@ -14,13 +14,23 @@ second service to run.
 pnpm add @cloudbitmaps/roaring @cloudbitmaps/gcs
 ```
 
-On npm 12 and pnpm 10 and later, allow `roaring`'s one install script first, or the install exits 0 and the package throws at `import`. Put this in your `package.json`: `{ "allowScripts": { "roaring": true }, "pnpm": { "onlyBuiltDependencies": ["roaring"] } }`. npm 11 runs the script but warns until you allow it the same way; pnpm 9 needs nothing extra. [Details](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/getting-started.md#cannot-find-module-buildreleaseroaringnode-after-a-successful-install).
+Install the same version of every `@cloudbitmaps/*` package: they release together, and each pins the exact version of `core` it was built with.
+
+`roaring` needs its one install script to fetch its native binary, and newer clients run a dependency's script only if you allow it. Allow it before you install:
+
+- **npm 12:** add `{ "allowScripts": { "roaring": true } }` to your `package.json`. Without it the install exits 0 and the package throws at `import`. npm 11 runs the script but warns until you allow it the same way.
+- **pnpm 11 and later:** put `allowBuilds: { roaring: true }` in a `pnpm-workspace.yaml` beside your `package.json`. Those versions no longer read the `package.json` `pnpm` key. Without it the install exits 1 with `ERR_PNPM_IGNORED_BUILDS`.
+- **pnpm 10:** add `{ "pnpm": { "onlyBuiltDependencies": ["roaring"] } }` to your `package.json`. Without it the install exits 0 and the package throws at `import`.
+- **pnpm 9:** nothing extra.
+
+[Details](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/getting-started.md#cannot-find-module-buildreleaseroaringnode-after-a-successful-install).
 
 `@google-cloud/storage` (`^7 || ^8`) is a real dependency of this package, so installing it is the whole step.
 `@cloudbitmaps/core`, the engine underneath, arrives with it and you never name it.
 
 ## Use
 
+<!-- typecheck -->
 ```ts
 import { CloudRoaring } from '@cloudbitmaps/roaring';
 import { GcsStorage } from '@cloudbitmaps/gcs';
@@ -49,10 +59,18 @@ It builds its own client from Application Default Credentials. Any other key is
 
 - **This backend is outside the validated envelope.** It passes the same conformance suites as S3, but the proven
   scale and the calibration were done on S3. You are an early user.
+- **Run `await store.preflight()` once before the first load.** It writes, tries to write again, reads, lists and deletes one probe
+  object, and reads the registry and the bucket's lifecycle rules (`storage.buckets.get` for the last), and answers
+  `{ ok, checks }`. It does not check a read after a delete or versioning, among others: [the full list](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/production.md#what-preflight-does-not-check). A first run is eight requests here.
 - **Grant the bucket's object permissions.** The library creates, reads, lists and deletes objects:
   `roles/storage.objectAdmin` covers all four.
 - **The write-once guarantee rides GCS preconditions** (`ifGenerationMatch`), so nothing else needs enabling.
 - **Never add a lifecycle rule that expires current objects or the `registry/` prefix.**
+- **Check the bucket's lifecycle rules with `await backend.checkLifecycle()`.** It reads the bucket's metadata with one
+  request (bounded by `readTimeoutMs`, when set) and answers `problem` for a `Delete` rule that applies to live objects and can match your prefix, `ok`, or
+  `unchecked`. It needs `storage.buckets.get`, which a load never uses, `store.preflight()`'s lifecycle step does, and `roles/storage.objectAdmin` does
+  not include: without it the answer is `unchecked`, naming the permission
+  ([check the lifecycle rules from code](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/production.md#check-the-lifecycle-rules-from-code)).
 - **The SDK does not retry downloads; the driver does.** In `@google-cloud/storage` 7.x and 8.x (checked on 7.22.0 and
   8.1.0), a download the SDK retries after any status it retries (408, 429, 500, 502, 503 or 504) can crash the process
   with `ERR_STREAM_UNABLE_TO_PIPE`, thrown outside any promise, even though the retried request succeeded. The client
@@ -98,17 +116,43 @@ It builds its own client from Application Default Credentials. Any other key is
   global agent closes a connection idle for 5 seconds (the SDK's own pool does not), so a read after a longer pause
   opens a new connection, with its TCP and TLS handshake, and downloads share that agent with any other `http` or
   `https` request in the process. Raising `https.globalAgent.options.timeout` keeps idle connections longer.
-- **A transient failure of a write throws `TransientError`, and the write may or may not have landed.** Re-run the
-  call, or check `store.generations(ref)`. The driver sends each registry write once; a load that gets no answer reads
-  the row and sends a fresh compare-and-swap from it, at most three times. A generation's object of up to
-  `simpleUploadThresholdBytes` is sent again, up to three more times, only after a `429` or `503`; every object carries
-  a random id in its metadata, so a first send that landed is told from another writer's object. A load whose row writes
-  all go unanswered throws `TransientError` and keeps its object. The waits before an object's re-sends use real
-  timers: the store's injected clock does not reach them, and `GcsStorage` takes no clock, so a test that throttles
-  one object write three times waits about 3.5 s at most (the publish's waits before a fresh row write do use the
-  store's clock).
+- **A transient failure of a write throws `TransientError`, and the write may or may not have landed.** Re-run the call,
+  or check `store.generations(ref)`. The driver [sends each registry write
+  once](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/correctness.md#sdk-retry); a load that gets no
+  answer [reads the row and sends a fresh compare-and-swap from
+  it](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/correctness.md#row-write-settled), at most three
+  times. A generation's object of up to `simpleUploadThresholdBytes` is sent again, up to three more times, only after a
+  `429` or `503`; every object carries a random id in its metadata, so a first send that landed is told from another
+  writer's object. A load whose row writes all go unanswered throws `TransientError` and keeps its object. The waits
+  before an object's re-sends use real timers: the store's injected clock does not reach them, and `GcsStorage` takes no
+  clock, so a test that throttles one object write three times waits about 3.5 s at most (the publish's waits before a
+  fresh row write do use the store's clock).
 
 The [production checklist](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/production.md) covers each item, and the ones every backend shares: backups of the data and the registry, and a schedule for your loads.
+- **The driver redacts the credentials it recognises from an error before it throws it.** The SDK keeps the response it
+  got on the error (`response`), and through it the request it sent, with the bearer token in its `Authorization`
+  header, and the HTTP agent it shares with other calls; a resumable upload's or a token request's error keeps the
+  request's `config`. On a download, and wherever a body lists no errors, the SDK makes the response body the error's
+  message, so a proxy's page or a JSON body that quotes the request puts the token in the message. In the error the
+  driver throws, the value under a credential's name is replaced by `[redacted]` (`authorization`,
+  `x-goog-encryption-key`, `x-goog-api-key`, `X-Goog-Signature`, `X-Goog-Credential`, `access_token`, `refresh_token`,
+  `client_secret`, `private_key` and the like; a service account's `client_email` and `private_key_id` are kept), and so
+  are the value after one in a header list or under one in a map, the value beside one in a record (a `{ name, value }`
+  pair, a `{ th, td }` row; a number, a boolean, and the `message`, `reason` and `location` of a Google error item are
+  kept), and, anywhere in its text, a credential header line; an `Authorization` value, an encryption key, an API key,
+  an OAuth token or client secret, or a signed URL's signature or credential after its name; a cookie header's value
+  after its name; a bearer token after its scheme; a Google access token (`ya29.`) or a JWT on its own; a V2 signed
+  URL's `Signature=`; a PEM private key's body; the text of an element named for a credential, such as the
+  `<CanonicalRequest>` a service quotes back; and the text of the cell after one that names a credential (a
+  `<th>`/`<td>` row, a `<dt>`/`<dd>` pair, after a heading or a caption too). A credential in another form is not
+  recognised (a header table laid out by column, a user and password written into a URL, a colon written as an HTML
+  entity such as `&colon;` in a body a download quotes as it came), and a text these steps cannot finish is replaced
+  whole. An SDK error that carries its transport or shows a credential, and every error it reaches (its `cause` chain,
+  an `AggregateError`'s `errors`), is thrown as a copy: a native error with the same prototype, name, message, stack,
+  `code`, `status` and `errors`, with `response` reduced to its status and status text, so classification is unchanged,
+  without `config`, the request, its agent or socket, any function or accessor property, or anything nested deeper than
+  ten levels. The live error is never changed, and an error that holds only plain data and shows no credential is thrown
+  as it is.
 
 ## Documentation
 

package/dist/backend.d.ts

@@ -1,20 +1,14 @@
-/**
- * `GcsStorage` — the Google Cloud Storage backend as one object: generations and pointer, one bucket, stated once.
- *
- * See {@link StorageBackend} for why the two halves are configured together rather than separately.
- *
- * **A note on the word.** `@google-cloud/storage` names its own client class `Storage`, so the driver option
- * that takes it is also called `storage` — which, since this library calls its durable tier storage too, reads
- * as the same word meaning two things. That is why this class builds the client itself: `new GcsStorage({ bucket })`
- * never makes you write `{ storage: storage }`. Supply `client` when you need your own.
- */
 import { STORAGE_BACKEND } from '@cloudbitmaps/core/driver-kit';
-import type { IRegistryDriver, IStorageDriver, StorageBackend } from '@cloudbitmaps/core/driver-kit';
+import type { IRegistryDriver, IStorageDriver, LifecycleVerdict, StorageBackend } from '@cloudbitmaps/core/driver-kit';
 import { Storage as GcsClient } from '@google-cloud/storage';
+/**
+ * What {@link GcsStorage} takes: a `bucket`, and either a `client` or the two settings that build one (`projectId`,
+ * `apiEndpoint`), never both.
+ */
 export interface GcsStorageOptions {
     /** Target bucket (must already exist). */
     readonly bucket: string;
-    /** Optional object-name prefix under which everything lives — generations and the registry alike. */
+    /** Optional object-name prefix under which everything lives — generations and the registry alike. A string, else `ValidationError`. */
     readonly prefix?: string;
     /**
      * A constructed `@google-cloud/storage` client. One is built from the ambient credentials when absent. Its retry
@@ -31,9 +25,15 @@ export interface GcsStorageOptions {
      * built, other than an interceptor, is not carried to the twin.
      */
     readonly client?: GcsClient;
-    /** Project id for the client built when `client` is absent (refused beside `client`). Falls back to the SDK's own resolution. */
+    /**
+     * Project id for the client built when `client` is absent (refused beside `client`): a non-empty string, else
+     * `ValidationError`. Falls back to the SDK's own resolution.
+     */
     readonly projectId?: string;
-    /** Endpoint override — point it at fake-gcs-server locally. Refused beside `client`, which carries its own. */
+    /**
+     * Endpoint override — point it at fake-gcs-server locally: a non-empty string, since an empty one would send
+     * requests to the public endpoint, else `ValidationError`. Refused beside `client`, which carries its own.
+     */
     readonly apiEndpoint?: string;
     /** Largest object the backend will write and advertise (default = GCS's 5 TiB max). Must be a positive safe integer. */
     readonly maxObjectBytes?: number;
@@ -77,6 +77,16 @@ export interface GcsStorageOptions {
  * for a client pointed at an emulator that is production traffic from a wiring typo.
  */
 export declare const GCS_STORAGE_OPTION_KEYS: readonly ["bucket", "prefix", "client", "projectId", "apiEndpoint", "maxObjectBytes", "simpleUploadThresholdBytes", "now", "readTimeoutMs", "conditionalDelete"];
+/**
+ * `GcsStorage` — the Google Cloud Storage backend as one object: generations and pointer, one bucket, stated once.
+ *
+ * See {@link StorageBackend} for why the two halves are configured together rather than separately.
+ *
+ * **A note on the word.** `@google-cloud/storage` names its own client class `Storage`, so the driver option
+ * that takes it is also called `storage` — which, since this library calls its durable tier storage too, reads
+ * as the same word meaning two things. That is why this class builds the client itself: `new GcsStorage({ bucket })`
+ * never makes you write `{ storage: storage }`. Supply `client` when you need your own.
+ */
 export declare class GcsStorage implements StorageBackend {
     /** Cross-bundle brand, stamped non-enumerably in the constructor so a spread cannot carry it. */
     readonly [STORAGE_BACKEND]: true;
@@ -84,5 +94,25 @@ export declare class GcsStorage implements StorageBackend {
     readonly registry: IRegistryDriver;
     /** The client both halves share — built here unless one was supplied. */
     readonly client: GcsClient;
+    private readonly bucket;
+    /** The name prefix every object sits under: `''`, or `trimmed/`. */
+    private readonly keyPrefix;
+    private readonly readTimeoutMs;
+    /**
+     * Builds the GCS backend over one bucket, which `bucket` names. Pass `client`, or `projectId` and `apiEndpoint` to
+     * have one built, not both. `prefix` puts every object under a name prefix. Throws `ValidationError` for an unknown
+     * option, no bucket, a `now` that is not a function, a client setting beside `client`, a `maxObjectBytes` or
+     * `simpleUploadThresholdBytes` that is not a positive integer, a `conditionalDelete` that is not a boolean, an
+     * empty or non-string `apiEndpoint` or `projectId`, or an invalid `prefix` or `readTimeoutMs`.
+     */
     constructor(options: GcsStorageOptions);
+    /**
+     * Reads the bucket's metadata (one request, plus the SDK's own retries, bounded by `readTimeoutMs` when it is set) and
+     * reports a lifecycle rule with a `Delete` action (or one it does not know) that applies to live
+     * objects whose names can be this store's: generations and registry rows alike. Needs `storage.buckets.get`, which
+     * a load never uses (`store.preflight()`'s lifecycle step does) and `roles/storage.objectAdmin` does not include: without it the answer is
+     * `unchecked`, naming the permission. Never rejects; a `detail` carries no credential. Rule reading: see the
+     * production guide.
+     */
+    checkLifecycle(): Promise<LifecycleVerdict>;
 }