@cloudbitmaps/roaring 0.14.0 → 0.16.0

package/PRIVACY.md

@@ -44,7 +44,7 @@ from constructing a cross-region topology. The points where personal data moves
 |---|---|---|
 | **Storage** (object store) | immutable `.crbm` generations — every generation a segment has had, until a superseded one is collected. Azure Blob objects, and GCS objects above the simple-upload threshold, also carry a random write id in their metadata (`cbwid`): 128 random bits, no data from the bitmap or the source | the region of the bucket you wire |
 | **Registry** (S3 / GCS / Azure Blob / local) | one row per segment: the current-generation pointer, wrapped keys, retention metadata, and the current generation's id count and your metadata (sealed when the segment is encrypted) — no IDs, unless you put one in the metadata, which you must not. An Azure Blob row also carries the random write id in its metadata | the region of the bucket you wire |
-| **cache** (process RAM) | decoded chunks, bounded LRU | **wherever your process/Lambda runs** — an EU segment queried from a US function is processed in the US |
+| **cache** (process RAM) | decoded chunks, and the stored chunk bytes (ciphertext on an encrypted segment) a reader keeps of a small generation it read whole; bounded LRU | **wherever your process/Lambda runs** — an EU segment queried from a US function is processed in the US |
 | **Loads and rewrites** (`store.load()`, the `*Into` verbs, `eraseSubject`) | read your source (or existing generations), write a new generation | run wherever you run them — a loader in one region writing to a bucket in another is a transfer |
 | **Intersection** | pulls chunks from N segments into one process | co-locates those segments in one region |
 
@@ -126,7 +126,7 @@ after it returns, and re-run any erasure a rollback interrupted.
 Erasure and crypto-shred are **immediate in storage and immediate in the store whose verb performed them**, for
 every read that starts after the verb returns. A read already in progress in that store can still yield an erased id
 from a chunk it had requested before the verb ran: up to 32 chunks for `iterate` and `count`, and up to `concurrency`
-keys (32 by default) for a combine. They are not immediate in any *other* store, in the same process or another, and this library ships nothing that could
+keys (32 by default) for a combine. A combine or `iterate` reads ranges of the object, and resolves the segment again before it serves each chunk, as a read of that chunk alone does, so the ranges it had already requested are dropped, not served. They are not immediate in any *other* store, in the same process or another, and this library ships nothing that could
 make them so — there is no daemon, no bus, and no connection between two stores that happen to point at the same
 bucket.
 
@@ -136,7 +136,7 @@ bucket.
 | the store whose verb made the call (`eraseSubject`, `dropSegment`, `retireExpired`) | on return, for every read that starts after it — it invalidates what it cached, and its pins then fail; a read already in progress there can still yield it from a chunk it had requested before |
 | another store, with a registry and a `cache.genTtlMs` above 0 | within `cache.genTtlMs` (default 2 s), when its snapshot re-resolves, **while the registry can be read** (see below) |
 | another store with **no registry** (a bare `IStorageDriver`), with `cache: { genTtlMs: 0 }`, or on a storage source built with **no clock** | **no bound** — only when its caches happen to let the segment go, or something tells it |
-| a pinned handle (`seg.pin()`) in another store | **no bound** — until that store's reader cache evicts the pin's reader and its chunk cache evicts the chunks the pin decoded, or something tells it |
+| a pinned handle (`seg.pin()`) in another store | **no bound** — until that store's reader cache evicts the pin's reader (in a store with a timed pointer refresh, a small generation's reader holds all of its chunks, decoded or not) and its chunk cache evicts the chunks the pin decoded, or something tells it |
 
 **An outage of the registry extends that bound.** A refresh that cannot read the row because of a transient fault
 (throttling, a 5xx, a dropped connection) keeps serving the generation the reader already holds, and the key it

package/README.md

@@ -60,7 +60,7 @@ name.
 | Option | What it holds |
 |---|---|
 | `storage` (required) | a backend (see above) |
-| `cache` | `maxChunks`, `genTtlMs` (how soon a reader sees a new load, default 2000 ms), `readerMax`, `readerMaxBytes` |
+| `cache` | `maxChunks`, `ttlMs` (an optional age limit on a cached chunk; omitted, a chunk stays until evicted), `genTtlMs` (how soon a reader sees a new load, default 2000 ms), `readerMax`, `readerMaxBytes` |
 | `encryption` | `{ keystore, required }`: encrypt segments at rest with your own key |
 | `retry` | tune the retry of reads, or `false` to turn it off |
 | `metrics` | a metrics sink; off by default |
@@ -69,10 +69,11 @@ name.
 
 ## What it costs
 
-Measured against real S3 in `us-east-1`, from inside the region, with the pointer in the same bucket as the data: the median cold `intersect`
-of two 500,000-id segments sharing 100 of their 1,999 chunks made 204 GETs, **$81.60 per million**, requesting only
-the shared chunks, and took 290.06 ms at the median from inside the region (run of 2026-10-04, whose client had 50 sockets; the client this release builds allows 128, and its effect is not measured). A segment's first `store.load()`
-is **$11.60 per million**, pointer included, from its measured requests at list prices. A cold `count()` is one pointer read: it reads no payload and no object.
+A cold `intersect` of two 500,000-id segments sharing 100 of their 1,999 chunks made 6 GETs and took 95.03 ms at the median, **$2.40 per
+million** at list prices: measured on S3 in-region, in the run of 2026-10-04, whose client had 128 sockets. On a spread layout it reads more
+bytes than it needs, which cross-region can cost more than the requests it saves. A segment's first `store.load()` is
+expected at **$11.20 per million**, pointer included: counted by running the engine and priced at list, not yet measured on a cloud. A cold `count()` is one
+pointer read: it reads no payload and no object.
 
 The trade is stated plainly. A membership check that misses the cache costs a ranged GET against object storage,
 where an in-process store costs a memory read. If you need sub-millisecond answers on a working set that fits a

package/dist/bin/export-segments.js

@@ -33,6 +33,7 @@ import {
   estimateCost,
   groundedReport,
   loadSegment,
+  loadSegmentChunks,
   mapWithConcurrency,
   resolveBudget,
   resolvePerOpBudget,
@@ -460,7 +461,8 @@ function bitmapAsLoadInput(input) {
 var roaringCodec = {
   empty: () => SafeBitmap.empty(),
   fromValues: (values) => SafeBitmap.fromValues(values),
-  safeDeserialize: (bytes, maxBytes, options) => SafeBitmap.safeDeserialize(bytes, maxBytes, options)
+  safeDeserialize: (bytes, maxBytes, options) => SafeBitmap.safeDeserialize(bytes, maxBytes, options),
+  owns: (bitmap) => bitmap instanceof SafeBitmap
 };
 
 // src/system-clock.ts
@@ -885,7 +887,7 @@ var CloudRoaring = class _CloudRoaring {
     const metadata = options?.metadata;
     let result;
     try {
-      result = await loadSegment(dest, ids, deps, {
+      const loadOptions = {
         ...options?.allowEmpty === void 0 ? {} : { allowEmpty: options.allowEmpty },
         ...options?.guard === void 0 ? {} : { guard: options.guard },
         ...metadata === void 0 ? {} : { metadata },
@@ -898,7 +900,8 @@ var CloudRoaring = class _CloudRoaring {
         // as a side effect of adding a guard whose entire purpose is preventing data loss. Opt in with `keep`.
         keep: options?.keep ?? KEEP_EVERY_GENERATION,
         ...options?.audit === void 0 ? {} : { audit: options.audit }
-      });
+      };
+      result = ids instanceof CombineChunks ? await loadSegmentChunks(dest, ids.chunks, deps, loadOptions) : await loadSegment(dest, ids, deps, loadOptions);
     } finally {
       this.engine.invalidate(dest);
     }
@@ -1702,6 +1705,32 @@ var EMPTY_IDS = {
   async *batches() {
   }
 };
+var AS_IDS = {
+  none: EMPTY_IDS,
+  failing,
+  intersect: (engine, refs, opts) => withBatches(engine.intersect(refs, opts), () => engine.intersectBatches(refs, opts)),
+  union: (engine, refs, opts) => withBatches(engine.union(refs, opts), () => engine.unionBatches(refs, opts)),
+  andNot: (engine, base, refs, opts) => withBatches(engine.andNot(base, refs, opts), () => engine.andNotBatches(base, refs, opts))
+};
+var NO_CHUNKS = {
+  async *[Symbol.asyncIterator]() {
+  }
+};
+var CombineChunks = class {
+  constructor(chunks) {
+    this.chunks = chunks;
+  }
+  chunks;
+};
+var AS_CHUNKS = {
+  none: NO_CHUNKS,
+  failing: (err) => ({
+    [Symbol.asyncIterator]: () => ({ next: () => Promise.reject(err) })
+  }),
+  intersect: (engine, refs, opts) => engine.intersectChunks(refs, opts),
+  union: (engine, refs, opts) => engine.unionChunks(refs, opts),
+  andNot: (engine, base, refs, opts) => engine.andNotChunks(base, refs, opts)
+};
 var KEEP_EVERY_GENERATION = Number.MAX_SAFE_INTEGER;
 var makeSegment;
 var minting = false;
@@ -1993,23 +2022,27 @@ var Segment = class _Segment {
   /**
    * Chunk-skipping intersection: stream the ids in **this** segment AND every segment in `others`, ascending.
    * Fetches only the Storage chunks present in *all* operands (a key absent from any operand contributes nothing
-   * and is never downloaded), streaming under a bounded in-flight window — so the Storage footprint stays small
-   * (Lambda-friendly) regardless of segment size. Pass `concurrency` to tune that window (a positive integer, 32 by default).
+   * and is never downloaded), reading each operand's as coalesced ranges under a bounded in-flight window — so the Storage
+   * footprint stays small (Lambda-friendly) regardless of segment size. Pass `concurrency` to tune that window (the range
+   * requests held ahead per operand: a positive integer, 32 by default).
    * AND is commutative, so `a.intersect([b])` and `b.intersect([a])` yield the same ids. Pass `budget` to
    * override the store's per-op denial-of-wallet budget for this call (or `false` to lift it).
    */
   intersect(others, options) {
-    if (this.expired() || others.some((o) => o.expired())) return EMPTY_IDS;
+    return this.intersectAs(AS_IDS, others, options);
+  }
+  intersectAs(out, others, options) {
+    if (this.expired() || others.some((o) => o.expired())) return out.none;
     const exclude = this.liveExcludes(options);
     let engine;
     try {
       engine = this.combineEngine([this, ...others, ...exclude]) ?? this.engine;
     } catch (err) {
-      return failing(err);
+      return out.failing(err);
     }
     const refs = [this.ref, ...others.map((o) => o.ref)];
     const opts = this.refsIn(options, exclude);
-    return withBatches(engine.intersect(refs, opts), () => engine.intersectBatches(refs, opts));
+    return out.intersect(engine, refs, opts);
   }
   /**
    * Materialize `this ∩ others…` (minus `exclude`) as a **new generation of `dest`** — `dest`'s previous contents
@@ -2036,7 +2069,12 @@ var Segment = class _Segment {
     this.combineEngine([this, ...others, ...options?.exclude ?? []]);
     return this.timed(
       "intersectInto",
-      () => this.materialize(dest.ref, this.intersect(others, options), "intersectInto", options)
+      () => this.materialize(
+        dest.ref,
+        new CombineChunks(this.intersectAs(AS_CHUNKS, others, options)),
+        "intersectInto",
+        options
+      )
     );
   }
   /**
@@ -2050,26 +2088,29 @@ var Segment = class _Segment {
    * segment once (`unionInto`, or a load) is the cheaper shape.
    */
   union(others, options) {
+    return this.unionAs(AS_IDS, others, options);
+  }
+  unionAs(out, others, options) {
     const live = others.filter((o) => !o.expired());
     if (this.expired()) {
-      if (live.length === 0) return EMPTY_IDS;
-      return live[0].union(live.slice(1), options);
+      if (live.length === 0) return out.none;
+      return live[0].unionAs(out, live.slice(1), options);
     }
     if (live.length === 0 && others.length > 0) {
       const exclude2 = this.liveExcludes(options);
-      return exclude2.length > 0 ? this.andNot([...exclude2], options) : this.union([], options);
+      return exclude2.length > 0 ? this.andNotAs(out, [...exclude2], options) : this.unionAs(out, [], options);
     }
-    if (live.length !== others.length) return this.union(live, options);
+    if (live.length !== others.length) return this.unionAs(out, live, options);
     const exclude = this.liveExcludes(options);
     let engine;
     try {
       engine = this.combineEngine([this, ...others, ...exclude]) ?? this.engine;
     } catch (err) {
-      return failing(err);
+      return out.failing(err);
     }
     const refs = [this.ref, ...others.map((o) => o.ref)];
     const opts = this.refsIn(options, exclude);
-    return withBatches(engine.union(refs, opts), () => engine.unionBatches(refs, opts));
+    return out.union(engine, refs, opts);
   }
   /** Materialize `this ∪ others…` (minus `exclude`) as a **new generation of `dest`** — see {@link intersectInto}. */
   async unionInto(dest, others, options) {
@@ -2077,7 +2118,12 @@ var Segment = class _Segment {
     this.combineEngine([this, ...others, ...options?.exclude ?? []]);
     return this.timed(
       "unionInto",
-      () => this.materialize(dest.ref, this.union(others, options), "unionInto", options)
+      () => this.materialize(
+        dest.ref,
+        new CombineChunks(this.unionAs(AS_CHUNKS, others, options)),
+        "unionInto",
+        options
+      )
     );
   }
   /**
@@ -2096,25 +2142,25 @@ var Segment = class _Segment {
    * the suppression folds into the same pass rather than materializing an intermediate segment first.
    */
   andNot(excludes, options) {
-    if (this.expired()) return EMPTY_IDS;
+    return this.andNotAs(AS_IDS, excludes, options);
+  }
+  andNotAs(out, excludes, options) {
+    if (this.expired()) return out.none;
     const liveExcludes = excludes.filter((e) => !e.expired());
     if (liveExcludes.length === 0 && excludes.length > 0) {
-      return this.union([], options == null ? void 0 : readOptions(options));
+      return this.unionAs(out, [], options == null ? void 0 : readOptions(options));
     }
-    if (liveExcludes.length !== excludes.length) return this.andNot(liveExcludes, options);
+    if (liveExcludes.length !== excludes.length) return this.andNotAs(out, liveExcludes, options);
     let engine;
     try {
       engine = this.combineEngine([this, ...excludes]) ?? this.engine;
     } catch (err) {
-      return failing(err);
+      return out.failing(err);
     }
     const base = this.ref;
     const refs = excludes.map((o) => o.ref);
     const opts = options == null ? void 0 : readOptions(options);
-    return withBatches(
-      engine.andNot(base, refs, opts),
-      () => engine.andNotBatches(base, refs, opts)
-    );
+    return out.andNot(engine, base, refs, opts);
   }
   /** Materialize `this \ (excludes…)` as a **new generation of `dest`** — see {@link intersectInto}. */
   async andNotInto(dest, excludes, options) {
@@ -2122,7 +2168,12 @@ var Segment = class _Segment {
     this.combineEngine([this, ...excludes]);
     return this.timed(
       "andNotInto",
-      () => this.materialize(dest.ref, this.andNot(excludes, options), "andNotInto", options)
+      () => this.materialize(
+        dest.ref,
+        new CombineChunks(this.andNotAs(AS_CHUNKS, excludes, options)),
+        "andNotInto",
+        options
+      )
     );
   }
   /**