@cloudbitmaps/roaring 0.13.0 → 0.15.0

package/PRIVACY.md

@@ -44,7 +44,7 @@ from constructing a cross-region topology. The points where personal data moves
 |---|---|---|
 | **Storage** (object store) | immutable `.crbm` generations — every generation a segment has had, until a superseded one is collected. Azure Blob objects, and GCS objects above the simple-upload threshold, also carry a random write id in their metadata (`cbwid`): 128 random bits, no data from the bitmap or the source | the region of the bucket you wire |
 | **Registry** (S3 / GCS / Azure Blob / local) | one row per segment: the current-generation pointer, wrapped keys, retention metadata, and the current generation's id count and your metadata (sealed when the segment is encrypted) — no IDs, unless you put one in the metadata, which you must not. An Azure Blob row also carries the random write id in its metadata | the region of the bucket you wire |
-| **cache** (process RAM) | decoded chunks, bounded LRU | **wherever your process/Lambda runs** — an EU segment queried from a US function is processed in the US |
+| **cache** (process RAM) | decoded chunks, and the stored chunk bytes (ciphertext on an encrypted segment) a reader keeps of a small generation it read whole; bounded LRU | **wherever your process/Lambda runs** — an EU segment queried from a US function is processed in the US |
 | **Loads and rewrites** (`store.load()`, the `*Into` verbs, `eraseSubject`) | read your source (or existing generations), write a new generation | run wherever you run them — a loader in one region writing to a bucket in another is a transfer |
 | **Intersection** | pulls chunks from N segments into one process | co-locates those segments in one region |
 
@@ -126,7 +126,7 @@ after it returns, and re-run any erasure a rollback interrupted.
 Erasure and crypto-shred are **immediate in storage and immediate in the store whose verb performed them**, for
 every read that starts after the verb returns. A read already in progress in that store can still yield an erased id
 from a chunk it had requested before the verb ran: up to 32 chunks for `iterate` and `count`, and up to `concurrency`
-keys (32 by default) for a combine. They are not immediate in any *other* store, in the same process or another, and this library ships nothing that could
+keys (32 by default) for a combine. A combine or `iterate` reads ranges of the object, and resolves the segment again before it serves each chunk, as a read of that chunk alone does, so the ranges it had already requested are dropped, not served. They are not immediate in any *other* store, in the same process or another, and this library ships nothing that could
 make them so — there is no daemon, no bus, and no connection between two stores that happen to point at the same
 bucket.
 
@@ -136,7 +136,7 @@ bucket.
 | the store whose verb made the call (`eraseSubject`, `dropSegment`, `retireExpired`) | on return, for every read that starts after it — it invalidates what it cached, and its pins then fail; a read already in progress there can still yield it from a chunk it had requested before |
 | another store, with a registry and a `cache.genTtlMs` above 0 | within `cache.genTtlMs` (default 2 s), when its snapshot re-resolves, **while the registry can be read** (see below) |
 | another store with **no registry** (a bare `IStorageDriver`), with `cache: { genTtlMs: 0 }`, or on a storage source built with **no clock** | **no bound** — only when its caches happen to let the segment go, or something tells it |
-| a pinned handle (`seg.pin()`) in another store | **no bound** — until that store's reader cache evicts the pin's reader and its chunk cache evicts the chunks the pin decoded, or something tells it |
+| a pinned handle (`seg.pin()`) in another store | **no bound** — until that store's reader cache evicts the pin's reader (in a store with a timed pointer refresh, a small generation's reader holds all of its chunks, decoded or not) and its chunk cache evicts the chunks the pin decoded, or something tells it |
 
 **An outage of the registry extends that bound.** A refresh that cannot read the row because of a transient fault
 (throttling, a 5xx, a dropped connection) keeps serving the generation the reader already holds, and the key it

package/README.md

@@ -60,7 +60,7 @@ name.
 | Option | What it holds |
 |---|---|
 | `storage` (required) | a backend (see above) |
-| `cache` | `maxChunks`, `genTtlMs` (how soon a reader sees a new load, default 2000 ms), `readerMax`, `readerMaxBytes` |
+| `cache` | `maxChunks`, `ttlMs` (an optional age limit on a cached chunk; omitted, a chunk stays until evicted), `genTtlMs` (how soon a reader sees a new load, default 2000 ms), `readerMax`, `readerMaxBytes` |
 | `encryption` | `{ keystore, required }`: encrypt segments at rest with your own key |
 | `retry` | tune the retry of reads, or `false` to turn it off |
 | `metrics` | a metrics sink; off by default |
@@ -69,10 +69,12 @@ name.
 
 ## What it costs
 
-Measured against real S3 in `us-east-1`, from inside the region, with the pointer in the same bucket as the data: the median cold `intersect`
-of two 500,000-id segments sharing 100 of their 1,999 chunks made 204 GETs, **$81.60 per million**, requesting only
-the shared chunks, and took 492.69 ms from inside the region, by an engine whose combine window held 8 chunk keys (this release opens it 8 wide and widens it to 32, so fewer rounds are expected; not yet measured). A segment's first `store.load()`
-is **$11.60 per million**, pointer included, from its measured requests at list prices. A cold `count()` is one pointer read: it reads no payload and no object.
+A cold `intersect` of two 500,000-id segments sharing 100 of their 1,999 chunks is expected to make 6 GETs, **$2.40 per
+million**: counted by running the engine and priced at list, not yet measured on a cloud. On a spread layout it reads more
+bytes than it needs, which cross-region can cost more than the requests it saves. A segment's first `store.load()` is
+**$11.60 per million**, pointer included, from its measured requests at list prices (run of 2026-10-04, in-region, whose
+client had 50 sockets; the client this release builds allows 128, and its effect is not measured). A cold `count()` is one
+pointer read: it reads no payload and no object.
 
 The trade is stated plainly. A membership check that misses the cache costs a ranged GET against object storage,
 where an in-process store costs a memory read. If you need sub-millisecond answers on a working set that fits a

package/dist/bin/export-segments.js

@@ -449,6 +449,10 @@ var SafeBitmap = class _SafeBitmap {
   toArray() {
     return this.bitmap.toArray();
   }
+  /** The values as one fresh ascending `Uint32Array`, exported natively. */
+  toUint32Array() {
+    return this.bitmap.toUint32Array();
+  }
 };
 function bitmapAsLoadInput(input) {
   return input instanceof RoaringBitmap32 ? { bitmap: input } : input;
@@ -1668,9 +1672,14 @@ var CloudRoaring = class _CloudRoaring {
     return estimateCost(input);
   }
 };
-var failing = (err) => ({
-  [Symbol.asyncIterator]: () => ({ next: () => Promise.reject(err) })
-});
+var withBatches = (gen, batches) => Object.assign(gen, { batches });
+var failing = (err) => {
+  const rejecting = () => ({ next: () => Promise.reject(err) });
+  return {
+    [Symbol.asyncIterator]: rejecting,
+    batches: () => ({ [Symbol.asyncIterator]: rejecting })
+  };
+};
 var samePin = (a, b) => a === void 0 || b === void 0 ? a === b : a.generation === b.generation && a.fingerprint === b.fingerprint;
 var twoPins = (a, b) => {
   const one = (p) => p === void 0 ? "live" : `pinned at generation ${p.generation ?? "none"}`;
@@ -1689,6 +1698,8 @@ var readOptions = (options) => ({
 var NO_SEGMENTS = [];
 var EMPTY_IDS = {
   async *[Symbol.asyncIterator]() {
+  },
+  async *batches() {
   }
 };
 var KEEP_EVERY_GENERATION = Number.MAX_SAFE_INTEGER;
@@ -1944,7 +1955,9 @@ var Segment = class _Segment {
     if (this.expired()) return EMPTY_IDS;
     const range = options == null ? void 0 : rangeOf(options);
     const none = range === void 0 || range.after === void 0 && range.through === void 0;
-    return this.engine.iterate(this.ref, none ? void 0 : range);
+    const { engine, ref } = this;
+    const bounds = none ? void 0 : range;
+    return withBatches(engine.iterate(ref, bounds), () => engine.iterateBatches(ref, bounds));
   }
   /**
    * The options a combine hands the engine, read once, when it is called ({@link readOptions}), with the `exclude`
@@ -1980,8 +1993,9 @@ var Segment = class _Segment {
   /**
    * Chunk-skipping intersection: stream the ids in **this** segment AND every segment in `others`, ascending.
    * Fetches only the Storage chunks present in *all* operands (a key absent from any operand contributes nothing
-   * and is never downloaded), streaming under a bounded in-flight window — so the Storage footprint stays small
-   * (Lambda-friendly) regardless of segment size. Pass `concurrency` to tune that window (a positive integer, 32 by default).
+   * and is never downloaded), reading each operand's as coalesced ranges under a bounded in-flight window — so the Storage
+   * footprint stays small (Lambda-friendly) regardless of segment size. Pass `concurrency` to tune that window (the range
+   * requests held ahead per operand: a positive integer, 32 by default).
    * AND is commutative, so `a.intersect([b])` and `b.intersect([a])` yield the same ids. Pass `budget` to
    * override the store's per-op denial-of-wallet budget for this call (or `false` to lift it).
    */
@@ -1994,7 +2008,9 @@ var Segment = class _Segment {
     } catch (err) {
       return failing(err);
     }
-    return engine.intersect([this.ref, ...others.map((o) => o.ref)], this.refsIn(options, exclude));
+    const refs = [this.ref, ...others.map((o) => o.ref)];
+    const opts = this.refsIn(options, exclude);
+    return withBatches(engine.intersect(refs, opts), () => engine.intersectBatches(refs, opts));
   }
   /**
    * Materialize `this ∩ others…` (minus `exclude`) as a **new generation of `dest`** — `dest`'s previous contents
@@ -2052,7 +2068,9 @@ var Segment = class _Segment {
     } catch (err) {
       return failing(err);
     }
-    return engine.union([this.ref, ...others.map((o) => o.ref)], this.refsIn(options, exclude));
+    const refs = [this.ref, ...others.map((o) => o.ref)];
+    const opts = this.refsIn(options, exclude);
+    return withBatches(engine.union(refs, opts), () => engine.unionBatches(refs, opts));
   }
   /** Materialize `this ∪ others…` (minus `exclude`) as a **new generation of `dest`** — see {@link intersectInto}. */
   async unionInto(dest, others, options) {
@@ -2091,10 +2109,12 @@ var Segment = class _Segment {
     } catch (err) {
       return failing(err);
     }
-    return engine.andNot(
-      this.ref,
-      excludes.map((o) => o.ref),
-      options == null ? void 0 : readOptions(options)
+    const base = this.ref;
+    const refs = excludes.map((o) => o.ref);
+    const opts = options == null ? void 0 : readOptions(options);
+    return withBatches(
+      engine.andNot(base, refs, opts),
+      () => engine.andNotBatches(base, refs, opts)
     );
   }
   /** Materialize `this \ (excludes…)` as a **new generation of `dest`** — see {@link intersectInto}. */