@cloudbitmaps/roaring 0.16.0 → 0.18.0

package/PRIVACY.md

@@ -121,6 +121,14 @@ ledger is not by itself proof the id is gone.
 generation the erasure did not rewrite, so the id can be current again. Roll back before the erasure starts or
 after it returns, and re-run any erasure a rollback interrupted.
 
+### A batch of materializations
+
+A `store.materializeMany` call computes many `*Into` outputs in passes that read each operand once per group, and its window is the call's length, not one read's. An output of a `materializeMany` call can carry an id that was erased while the call ran, for as long as the call ran: the call reads each operand at the generation it pinned, and an output's publish is a load, so an erasure that lands after the chunks were read does not reach what the call holds, unless the output subtracts a pinned operand that the erasure rewrote, which the call re-reads just before the publishes and refuses. An erasure that rewrites a destination before that output's publish starts does not stop the publish, which, subject to the output's own `guard` and the refusal of an empty result over a non-empty destination, writes on top of the erasure's generation; an erasure that lands inside the publish's own write, between its pointer read and its pointer write, makes the publish lose with `WriteConflictError`; after any call that overlapped an erasure, re-run `eraseSubject` and keep both ledgers. `eraseSubject` deletes the generation it rewrites even when a call holds it pinned, so a pin does not outlive an erasure, and an output that still needs a deleted operand generation fails with `NotFoundError`.
+
+A call with a feed reads operands the library never stores, which no erasure can reach, so the guarantee above does not describe it, and it is refused instead. A `materializeMany` call with a feed is refused at its next record, and immediately before each fed output's publish (after the output has waited for its room in the memory budget), once `eraseSubject` has started in this store, one already running when the call began included: nothing is published from operands the call read before the erasure returned, apart from the window below, and the one record in hand when the erasure lands is processed and discarded. The counter it reads is moved at the start and at the end of each `eraseSubject` this store runs, so an erasure that began and finished between two of the call's checks is seen too; `rollback`, `retireExpired` and `dropSegment` do not move it. An erasure that starts after a fed output's last check and finishes before that publish's pointer write is not caught by the counter, and its window is the publish's own object write and pointer write, which lasts as long as the object takes to upload; the fences above still apply to what it touched. Only this store's `eraseSubject` moves the counter such a call reads, so an erasure by any other path (another store, another process, or the free function `eraseIdFromSegment` in this process) is not seen by it, and is bounded by nothing. An id erased from the source of a feed has to be erased there: the library holds none of it.
+
+A held operand, made by `store.memory`, is the caller's memory too, so no erasure can reach it either, and it is refused instead. A `materializeMany` call that reads an in-memory operand from `store.memory` fails the outputs that read it, before any request and immediately before each of their publishes, once `eraseSubject` has started in this store since the operand began to be made, one already running when it began included; an erasure by any path but this store's `eraseSubject` is not seen by such an operand, and is bounded by nothing. An erasure that starts after an output's last check and finishes before its publish writes is not caught by the counter, as for a feed, and the fences above still apply to what it touched. Build a held operand from your source of truth at the start of each refresh and release it at its end: the library keeps none of it once released.
+
 ### One process, and the rest of your fleet
 
 Erasure and crypto-shred are **immediate in storage and immediate in the store whose verb performed them**, for
@@ -136,7 +144,11 @@ bucket.
 | the store whose verb made the call (`eraseSubject`, `dropSegment`, `retireExpired`) | on return, for every read that starts after it — it invalidates what it cached, and its pins then fail; a read already in progress there can still yield it from a chunk it had requested before |
 | another store, with a registry and a `cache.genTtlMs` above 0 | within `cache.genTtlMs` (default 2 s), when its snapshot re-resolves, **while the registry can be read** (see below) |
 | another store with **no registry** (a bare `IStorageDriver`), with `cache: { genTtlMs: 0 }`, or on a storage source built with **no clock** | **no bound** — only when its caches happen to let the segment go, or something tells it |
+| `seg.pinAt` of a generation the erasure rewrote, in any store | on return it fails with `NotFoundError`, since the object is deleted (and, with a registry, the row has moved on); during the erasure, between the rewrite's publish and its delete of the old generation, it can still open that generation, and a handle opened then is a pin, covered by the row below |
+| a `materializeMany` call with a feed, and an erasure by any path but this store's `eraseSubject` | **no bound**: the call is refused only by this store's `eraseSubject` |
+| a held in-memory operand (`store.memory`), and an erasure by any path but this store's `eraseSubject` | **no bound**: the operand is refused only by this store's `eraseSubject`, started after the operand began to be made |
 | a pinned handle (`seg.pin()`) in another store | **no bound** — until that store's reader cache evicts the pin's reader (in a store with a timed pointer refresh, a small generation's reader holds all of its chunks, decoded or not) and its chunk cache evicts the chunks the pin decoded, or something tells it |
+| a leased pinned handle (`seg.pin({ leaseUntil })`) in another store | until its lease ends, at most 14 days after it was taken: its reads then throw `LeaseExpiredError`, whether or not its reader still holds the chunks. Until then, as for any pinned handle in the row above |
 
 **An outage of the registry extends that bound.** A refresh that cannot read the row because of a transient fault
 (throttling, a 5xx, a dropped connection) keeps serving the generation the reader already holds, and the key it
@@ -161,6 +173,20 @@ yours.
 A read served from a stale cache is bounded by the same window and answers from memory, so nothing on the
 storage side — a bucket policy, a lifecycle rule, the object's own deletion — can shorten it.
 
+**A lease keeps superseded generations, and erasure ignores it.** A pinned handle taken with `pin({ leaseUntil })` keeps
+its generation out of a load's collection until the lease ends: at most 14 days after it was taken, and a 60-second margin
+after that. While it does, the bucket holds a generation that a newer load has superseded, including ids that load
+removed. `eraseSubject`, `eraseIdFromSegment`, `dropSegment`, `destroySegment` and a retention expiry never read a lease:
+they delete the generations they must, a leased one included, so a lease never holds an erased subject's data past the
+bounds above, and erasure, shred, drop and retention always win over it. A rewrite, a shred and a drop clear the row's
+leases; an erasure that deletes a generation without rewriting leaves an entry for it in the row, which spares nothing and
+is pruned by the next publish or lease write after it ends. A lease write does not delay an erasure: a row that differs
+from the one the erasure read only in its leases does not refuse it. What a lease holds against is an ordinary load's
+collection, and the generation is collected by the next listing pass after the lease ends: within 16 later loads of the
+segment, and a segment that is never loaded again keeps it until an erasure, a drop or a retention expiry takes it. A
+leased handle stops answering when its lease ends, and its reads then throw `LeaseExpiredError`. The lease entry in the
+registry row is a random holder id, a generation number and an instant, with no id and no count.
+
 **Your exit path** (and a building block for a **data-portability / Art. 20** response): `store.exportSegments(sink,
 { format })` (and the `export-segments` CLI) dumps every registered segment's current generation to a portable
 file — `roaring` (loadable by any roaring library) or `ndjson` (newline ids) — readable without CloudBitmaps.

package/README.md

@@ -69,10 +69,10 @@ name.
 
 ## What it costs
 
-A cold `intersect` of two 500,000-id segments sharing 100 of their 1,999 chunks made 6 GETs and took 95.03 ms at the median, **$2.40 per
-million** at list prices: measured on S3 in-region, in the run of 2026-10-04, whose client had 128 sockets. On a spread layout it reads more
-bytes than it needs, which cross-region can cost more than the requests it saves. A segment's first `store.load()` is
-expected at **$11.20 per million**, pointer included: counted by running the engine and priced at list, not yet measured on a cloud. A cold `count()` is one
+A cold `intersect` of two 500,000-id segments sharing 100 of their 1,999 chunks made 6 GETs and took 99.80 ms at the median, **$2.40 per
+million** at list prices: measured on S3 in-region, in the run of 2026-10-06, whose client had 128 sockets. On a spread layout it reads more
+bytes than it needs, which cross-region can cost more than the requests it saves. A segment's first `store.load()` made
+2 PUT + 3 GET, **$11.20 per million** at the default prices, pointer included: the requests are measured on S3 in-region, in the same run. A steady `store.load()` at `keep: 12` that deletes a generation by name made 2 PUT + 4 GET and a free delete, measured: **$11.60 per million**. A cold `count()` is one
 pointer read: it reads no payload and no object.
 
 The trade is stated plainly. A membership check that misses the cache costs a ranged GET against object storage,