@cloudbitmaps/roaring 0.11.1 → 0.12.0

package/PRIVACY.md

@@ -43,7 +43,7 @@ from constructing a cross-region topology. The points where personal data moves
 | Location | What's there | Residency note |
 |---|---|---|
 | **Storage** (object store) | immutable `.crbm` generations — every generation a segment has had, until a superseded one is collected. Azure Blob objects, and GCS objects above the simple-upload threshold, also carry a random write id in their metadata (`cbwid`): 128 random bits, no data from the bitmap or the source | the region of the bucket you wire |
-| **Registry** (S3 / GCS / Azure Blob / local) | one row per segment: the current-generation pointer, wrapped keys, retention metadata — no IDs. An Azure Blob row also carries the random write id in its metadata | the region of the bucket you wire |
+| **Registry** (S3 / GCS / Azure Blob / local) | one row per segment: the current-generation pointer, wrapped keys, retention metadata, and the current generation's id count and your metadata (sealed when the segment is encrypted) — no IDs, unless you put one in the metadata, which you must not. An Azure Blob row also carries the random write id in its metadata | the region of the bucket you wire |
 | **cache** (process RAM) | decoded chunks, bounded LRU | **wherever your process/Lambda runs** — an EU segment queried from a US function is processed in the US |
 | **Loads and rewrites** (`store.load()`, the `*Into` verbs, `eraseSubject`) | read your source (or existing generations), write a new generation | run wherever you run them — a loader in one region writing to a bucket in another is a transfer |
 | **Intersection** | pulls chunks from N segments into one process | co-locates those segments in one region |
@@ -60,8 +60,8 @@ CloudBitmaps gives you three levers with different guarantees. Use them delibera
 
 | Lever | API | Guarantee | Use for |
 |---|---|---|---|
-| **Subject erasure** | `store.eraseSubject(id, { namespace })` | *Physical on return* — the segment's current generation is rewritten without the id (every chunk streamed through, one bit cleared), published **fenced on the generation it streamed**, and **the generation that held the bit is deleted before the call returns**. So is every other generation still holding the id: a retained *superseded* one (an ex-member dropped by a re-seed), and each one above the pointer after a `rollback` — generations a rollback can move the pointer onto again — while, when the current generation does not hold the id, the ones there that never held it stay as rollback targets (a rewrite is numbered above everything, so its `keep: 0` collection takes every older generation, above the pointer or below). Before it says `erased: true` the call lists the bucket and reads what is left, so **`erased: true` means no generation of the segment holds the id**, above the pointer or below it. `eraseSubject` reports a per-segment fault as an `error: …` ledger entry rather than throwing it, and a call that a racing writer overtook (a load, another erasure, or a rollback) as `erased: false, note: 'superseded'`, so "on return" is a claim about every segment whose entry says **`erased: true`** (a fault is an `error: …` entry, never `erased: true`). The store that performs the call stops serving the id immediately; **other processes converge on their own read TTL** — see [One process, and the rest of your fleet](#one-process-and-the-rest-of-your-fleet). Does **not** reach backups, replicas or noncurrent object versions — those hold the old generation until their own lifecycle removes it. | "forget this person" — GDPR Art. 17 |
-| **Dispose** | `store.dropSegment(ref, { confirmSegment })` | *Immediate* — the segment is tombstoned and its Storage generations deleted, reclaiming the space. **Check `generationsRemaining`:** if it is non-empty the space was *not* fully reclaimed and the drop should be re-run (a load that was already writing when the tombstone landed still finishes its object). Works on cleartext; on an encrypted segment it *also* crypto-shreds, on the terms in the next row. Does **not** reach noncurrent versions / replicas / PITR snapshots — deleting an object is weaker than removing a key. | retiring a dated bucket; rolling-window retention |
+| **Subject erasure** | `store.eraseSubject(id, { namespace })` | *Physical on return* — the segment's current generation is rewritten without the id (every chunk streamed through, one bit cleared), published **fenced on the generation it streamed**, and **the generation that held the bit is deleted before the call returns**. So is every other generation still holding the id: a retained *superseded* one (an ex-member dropped by a re-seed), and each one above the pointer after a `rollback` — generations a rollback can move the pointer onto again — while, when the current generation does not hold the id, the ones there that never held it stay as rollback targets (a rewrite is numbered above everything, so its `keep: 0` collection takes every older generation, above the pointer or below). Before it says `erased: true` the call lists the bucket and reads what is left, so **`erased: true` means no generation of the segment holds the id**, above the pointer or below it. `eraseSubject` reports a per-segment fault as an `error: …` ledger entry rather than throwing it, and a call that a racing writer overtook (a load, another erasure, or a rollback) as `erased: false, note: 'superseded'`, so "on return" is a claim about every segment whose entry says **`erased: true`** (a fault is an `error: …` entry, never `erased: true`). The store that performs the call stops serving the id immediately; **other processes converge on their own read TTL** — see [One process, and the rest of your fleet](#one-process-and-the-rest-of-your-fleet). Does **not** reach backups, replicas or noncurrent object versions — those hold the old generation until their own lifecycle removes it. It carries a generation's metadata over unchanged and does **not** scan it, so **never put a subject's id in metadata**. | "forget this person" — GDPR Art. 17 |
+| **Dispose** | `store.dropSegment(ref, { confirmSegment })` | *Immediate* — the segment is tombstoned and its Storage generations deleted, reclaiming the space. **Check `generationsRemaining`:** if it is non-empty the space was *not* fully reclaimed and the drop should be re-run (a load that had read the segment before the tombstone landed, even one still consuming its ids, can write its object after the drop's sweep; it deletes that object itself once its publish is refused, and only one whose process stops in between, or whose publish fails without a definite answer (a lost response, a timeout), leaves it, for a re-run of the drop). Works on cleartext; on an encrypted segment it *also* crypto-shreds, on the terms in the next row. Does **not** reach noncurrent versions / replicas / PITR snapshots — deleting an object is weaker than removing a key. | retiring a dated bucket; rolling-window retention |
 | **Crypto-shred** | `destroySegment` / `eraseNamespace` | *At rest, complete once no copy of the wrapped key survives* — removes the segment's wrapped DEK(s) from its registry row in one compare-and-swap, without touching the bytes or any KEK. **Every** copy of the objects (current, prior generations, backups, WORM-locked objects) is then unreadable once no retained copy of that row still holds the wrapped key — noncurrent object versions, backups and PITR copies of the row included — or once every KEK that wrapped it is destroyed. Until then, a retained copy of the row and a KEK that wrapped it decrypt the segment, and a registry restore to a point before the shred makes it readable again; see *Erasure vs. backups / WORM* below, and [an optional registry expiry rule](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/disaster-recovery.md#optional-make-a-shred-durable-with-a-registry-expiry-rule) that makes a shred durable after a set number of days. Requires the segment to be encrypted. A process that had already opened the segment holds the **unwrapped** key in memory and keeps reading until it is told — call `store.invalidate(ref)` there; see [One process, and the rest of your fleet](#one-process-and-the-rest-of-your-fleet). | whole-segment / tenant offboarding; erasure under immutable backups (see below) |
 
 **Subject-wide erasure** (GDPR Art. 17 — "forget this person everywhere") is
@@ -123,15 +123,17 @@ after it returns, and re-run any erasure a rollback interrupted.
 
 ### One process, and the rest of your fleet
 
-Erasure and crypto-shred are **immediate in storage and immediate in the store whose verb performed them**. They
-are not immediate in any *other* store, in the same process or another, and this library ships nothing that could
+Erasure and crypto-shred are **immediate in storage and immediate in the store whose verb performed them**, for
+every read that starts after the verb returns. A read already in progress in that store can still yield an erased id
+from a chunk it had requested before the verb ran: up to 8 chunks for `iterate` and `count`, and up to `concurrency`
+keys for a combine. They are not immediate in any *other* store, in the same process or another, and this library ships nothing that could
 make them so — there is no daemon, no bus, and no connection between two stores that happen to point at the same
 bucket.
 
 | | when the id stops being readable |
 |---|---|
 | storage | on return — the generation holding it is deleted, or the wrapped DEK is removed from the segment's current registry row (a crypto-shred is complete once no retained copy of that row holds it, or every KEK that wrapped it is destroyed; see *Erasure vs. backups / WORM* below) |
-| the store whose verb made the call (`eraseSubject`, `dropSegment`, `retireExpired`) | on return — it invalidates what it cached, and its pins then fail |
+| the store whose verb made the call (`eraseSubject`, `dropSegment`, `retireExpired`) | on return, for every read that starts after it — it invalidates what it cached, and its pins then fail; a read already in progress there can still yield it from a chunk it had requested before |
 | another store, with a registry and a `cache.genTtlMs` above 0 | within `cache.genTtlMs` (default 2 s), when its snapshot re-resolves, **while the registry can be read** (see below) |
 | another store with **no registry** (a bare `IStorageDriver`), with `cache: { genTtlMs: 0 }`, or on a storage source built with **no clock** | **no bound** — only when its caches happen to let the segment go, or something tells it |
 | a pinned handle (`seg.pin()`) in another store | **no bound** — until that store's reader cache evicts the pin's reader and its chunk cache evicts the chunks the pin decoded, or something tells it |
@@ -242,6 +244,16 @@ to run, and the deletion is ours to perform correctly.** Practical patterns:
   assumed: `generationsRemaining` non-empty means bytes survived and the call should be repeated. It tombstones
   the segment and deletes **every Storage generation** — so the space is actually reclaimed. It works on a
   cleartext segment, and on an encrypted one it *also* crypto-shreds, making it a strict superset there.
+- **A deleted row keeps its record only where the registry cannot remove it.** Where the registry reports
+  `conditionalDelete` (the default for S3 when its client sends to an AWS S3 host, for
+  Azure Blob, the local filesystem and memory), `registry.delete` and the retention sweep's purge of a tombstone remove
+  a row created by 0.12 from the bucket, and its token's random incarnation id keeps a re-create apart from it. Where it
+  does not (an S3 client that sends to an S3-compatible store or an emulator, or a GCS client, by default, or
+  `conditionalDelete: false`), and for a row written by a release before 0.12 on any backend, a delete writes a deleted marker that holds
+  the record, so the row's token counter survives a re-create. A tombstone left by `dropSegment` or `destroySegment`
+  holds no wrapped key and no summary, since both clear them, so a purge that leaves a tombstone keeps the name, the
+  pointer, the retention policy and the timestamps. A **live** row deleted directly with `registry.delete` keeps its
+  summary, the current generation's id count and your metadata, in a marker where one is written.
 - **`destroySegment` crypto-shreds** — it removes the wrapped key from the registry row, so the Storage bytes
   become unreadable *everywhere including backups, replicas and WORM-locked copies* once no retained copy of that
   row still holds the key, or every KEK that wrapped it is destroyed (see the trap above), which no object
@@ -264,16 +276,41 @@ So the two are complements, not alternatives: **`dropSegment` for "stop paying f
 > failed `dropSegment`, and a fine way to expire *noncurrent* object versions; give it a window comfortably
 > longer than your retention and never let it touch a current generation.
 
-> ⚠️ **Never let a lifecycle rule expire a registry row's current version or its tombstone** — a separate trap,
-> and a worse one. Deleting a registry row does not remove it: it writes a **tombstone** carrying a counter that only
-> ever advances, which is what makes the row's token unique for all time. Expire those tombstones and a segment
-> re-created under the same name starts the counter again and re-issues a token that was already used. That
-> token is the segment's **identity**: it is what a cached reader, a fenced publish and a generation-collection
-> pass each compare to decide whether two observations describe the same segment. Re-issue one and they can all
-> answer "yes" about a segment that no longer exists — serving a deleted incarnation's data, or collecting a
-> live one's objects. Storage for a tombstone is a few bytes per retired segment; treat it as permanent. A rule
-> on *noncurrent* versions of the registry prefix touches neither the current row nor its tombstone; what it
-> shortens is the history a registry restore picks from, and the time a crypto-shred takes to complete (above).
+> ⚠️ **Never let a lifecycle rule expire a registry row's current version, a tombstone, or a due-index pointer** —
+> a separate trap, and a worse one. A rule cannot tell a live row from a tombstone, and an expired live row is a
+> segment whose pointer is gone: its generations look unreferenced, and nothing reads or collects them. The library
+> removes a row only by a delete the store applies to the exact version it judged, and only a row whose token
+> carries a random incarnation id, so a name re-created later is told apart from it with overwhelming probability.
+> A tombstone of a row written by a release before 0.12 carries the row's token counter on instead, and that counter
+> is what keeps the name's tokens unique against a process still on that release. That token is the segment's
+> **identity**: it is what a cached reader, a fenced publish and a generation-collection pass each compare to decide
+> whether two observations describe the same segment. Re-issue one and they can all answer "yes" about a segment that
+> no longer exists — serving a deleted incarnation's data, or collecting a live one's objects. A rule on
+> *noncurrent* versions of the registry prefix touches neither the current row nor its tombstone; what it shortens is
+> the history a registry restore picks from, and the time a crypto-shred takes to complete (above).
+
+**What the registry keeps of a retired segment, and for how long.** A retirement by `retireExpired` leaves a
+`destroyed` row: the segment's namespace and name, its `retention` and `residency` metadata (any keys of yours in
+them included), its timestamps and its token. It holds no id, and on an encrypted segment no wrapped key, since the
+drop shreds it. The due index holds a pointer to it whose name spells out the same namespace and name. Both stay
+for `tombstoneGraceMs` (24 h by default) after the retirement, and for as long as the segment's storage cannot be
+proven gone. Then the sweep's purge removes the row, and the pointers it read to it, from the bucket, where the registry
+reports `conditionalDelete`: the default for S3 when its client sends to an AWS S3 host,
+for Azure Blob, the local filesystem and memory. **A pointer can outlive its row**, and with it the name in its key: a
+purge that ran with another `tombstoneGraceMs` than the sweep that filed the pointer, a delete that landed and lost its
+response, a removal the registry refused (`purgeFaults` counts it), or a pointer older than an index scan's
+`lookbackBuckets`. The next sweep that reads it removes it once its row is confirmed absent: an index scan from the days
+it reads, and an unscoped fleet scan, one given no `namespace`, from every day. A deployment that scopes every sweep to a
+namespace, or runs only index scans, keeps such a pointer until an unscoped fleet scan runs. Where the registry does not
+report `conditionalDelete` (an S3 client that sends to an S3-compatible store or an emulator, or a GCS client, by default, or
+`conditionalDelete: false`), and for a row written by a release before 0.12 on any backend, the purge leaves a tombstone
+instead: every one of those fields, in the bucket, indefinitely, read by every full listing. A registry that does not
+report it files no pointer to begin with. A removed
+row stays recoverable wherever the storage keeps a copy of it: with object versioning on, its earlier versions stay until
+a noncurrent-version rule expires them, as an overwritten row's do; with soft delete on, it stays for the retention
+window (GCS: on by default for a new bucket, 7 days; Azure Blob: where enabled, for the days set), whatever the registry
+did. A tombstone a hand-run `dropSegment` or a
+crypto-shred left is never purged by the library.
 
 Full detail, including the dated-bucket pattern and the pitfalls, is in the retention section of the
 [retention guide](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/retention.md).

package/README.md

@@ -48,7 +48,9 @@ To persist, pass a different **backend** as `storage`. Nothing else changes:
 | `AzureBlobStorage({ connectionString, container })` | `@cloudbitmaps/azure-blob` | Azure Blob Storage; outside the [validated envelope](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/ROADMAP.md#the-validated-envelope--whats-proven-and-what-isnt) |
 
 There is no `add` or `remove`: a segment changes only by loading a new generation. A load is a batch job, not a
-request handler. `store.exists(ref)` says whether a segment has been loaded.
+request handler. `store.exists(ref)` says whether a segment has been loaded. A load takes ids, or a bitmap you already
+hold: `store.load(ref, { bitmap })` for a `RoaringBitmap32`, or `{ serialized }` for portable Roaring bytes, checked
+before anything is written and loaded with no per-id work, into the same bytes its ids would write.
 
 ## Options
 
@@ -70,7 +72,7 @@ name.
 Measured against real S3 in `us-east-1`, with the pointer in the same bucket as the data: the median cold `intersect`
 of two 500,000-id segments sharing 100 of their 1,999 chunks made 206 GETs, **$82.40 per million**, requesting only
 the shared chunks. Inside the region it is expected at 204 GETs, $81.60 per million. Writing and publishing a segment
-is **$11.20 per million**, pointer included, from its measured requests at list prices. `count()` reads no payload.
+is **$11.20 per million**, pointer included, from its measured requests at list prices. A cold `count()` is one pointer read: it reads no payload and no object.
 
 The trade is stated plainly. A membership check that misses the cache costs a ranged GET against object storage,
 where an in-process store costs a memory read. If you need sub-millisecond answers on a working set that fits a
@@ -80,8 +82,8 @@ give the method and what the numbers do not establish.
 ## Before production
 
 - Grant the permissions the library issues and add a lifecycle rule to abort incomplete uploads.
-- Use a bucket that honors conditional writes, and `@aws-sdk/client-s3` 3.645.0 or later if you pass your own S3 client.
-- Set a request timeout on your storage client. Reads retry by themselves; a write that fails transiently is yours to re-run.
+- Use a bucket that honors conditional writes, and `@aws-sdk/client-s3` 3.700.0 or later if you pass your own S3 client.
+- Set a request timeout on your storage client. Reads retry by themselves, and a read can be timed on every backend (`readTimeoutMs`); a write that fails transiently is yours to re-run.
 - Back up the registry with the data, and the keystore too if you encrypt. If you lose the key, the data is gone.
 - Schedule your loads, and `retireExpired` if you use retention. Run loads off the request path.