@cloudbitmaps/roaring 0.9.0 → 0.11.0

package/NOTICE

@@ -1,4 +1,4 @@
-CloudRoaring
+@cloudbitmaps/roaring
 Copyright 2026 Sharvil Kadam
 
 This product includes software developed as an open-source project.
@@ -7,6 +7,6 @@ file. You may obtain a copy of the License at:
 
     http://www.apache.org/licenses/LICENSE-2.0
 
-CloudRoaring builds on the Roaring Bitmap data structure and is designed to use
-the CRoaring engine (via the `roaring` Node.js binding) for in-memory bit math.
+@cloudbitmaps/roaring builds on the Roaring Bitmap data structure and uses the CRoaring
+engine (via the `roaring` Node.js binding) for in-memory bit math.
 Those projects are independent works under their own licenses.

package/PRIVACY.md

@@ -1,5 +1,9 @@
 # Privacy, data protection & shared responsibility
 
+> This is the copy published with the npm package; it is kept identical in substance to
+> [`PRIVACY.md`](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/PRIVACY.md) in the repository.
+
+
 > **Engineering guidance, not legal advice.** This document explains CloudBitmaps' data-handling posture and
 > the controls it gives you, so you (and your privacy counsel / DPO) can place it correctly in your compliance
 > program. Regulatory references (GDPR, CCPA/CPRA, HIPAA, …) are illustrative. Nothing here is a compliance
@@ -8,7 +12,7 @@
 ## TL;DR — the trust boundary
 
 **CloudBitmaps is an embedded library.** It runs inside *your* process/function and talks to *your* storage
-accounts (S3, DynamoDB, local disk — whatever drivers you wire). **It has no hosted backend and transmits no
+accounts (S3, GCS, Azure Blob, local disk — whatever drivers you wire). **It has no hosted backend and transmits no
 data to the authors or any third party** — there is no telemetry, no phone-home, no usage ping. The metrics
 and audit sinks are local and off by default; they only go where *you* send them.
 
@@ -38,105 +42,293 @@ from constructing a cross-region topology. The points where personal data moves
 
 | Location | What's there | Residency note |
 |---|---|---|
-| **Cold** (object store) | immutable `.crbm` generations | the region of the bucket you wire |
-| **Warm** (NoSQL) | pending add/remove deltas | the region of the table you wire |
-| **HOT cache** (process RAM) | decoded chunks, bounded LRU | **wherever your process/Lambda runs** — an EU segment queried from a US function is processed in the US |
-| **Compaction** (daemon) | reads Cold+Warm, writes new Cold | runs wherever you run the daemon |
+| **Storage** (object store) | immutable `.crbm` generations — every generation a segment has had, until a superseded one is collected. Azure Blob objects, and GCS objects above the simple-upload threshold, also carry a random write id in their metadata (`cbwid`): 128 random bits, no data from the bitmap or the source | the region of the bucket you wire |
+| **Registry** (S3 / GCS / Azure Blob / local) | one row per segment: the current-generation pointer, wrapped keys, retention metadata — no IDs. An Azure Blob row also carries the random write id in its metadata | the region of the bucket you wire |
+| **cache** (process RAM) | decoded chunks, bounded LRU | **wherever your process/Lambda runs** — an EU segment queried from a US function is processed in the US |
+| **Loads and rewrites** (`store.load()`, the `*Into` verbs, `eraseSubject`) | read your source (or existing generations), write a new generation | run wherever you run them — a loader in one region writing to a bucket in another is a transfer |
 | **Intersection** | pulls chunks from N segments into one process | co-locates those segments in one region |
 
 **Guidance (not enforced by the library):** to keep EU data in EU infrastructure, wire region-local drivers
-*and* run the process/daemon in-region; keep a segment's Warm, Cold, and the querying compute in one
-jurisdiction; treat the HOT cache and the intersection runtime as **processing locations** in your transfer
-assessment and breach scope (process RAM, and any heap/core dumps, hold personal data). A fail-closed
-residency-enforcement policy in the library was considered and deferred as over-engineering for v1 — the honest
-posture is "you wire it correctly," documented here.
+*and* run your loaders and erasure jobs in-region; keep a segment's storage objects, its registry row, and the querying compute in
+one jurisdiction; treat the cache and the intersection runtime as **processing locations** in your transfer
+assessment and breach scope (process RAM, and any heap/core dumps, hold personal data). The library has no
+fail-closed residency-enforcement policy; the posture is "you wire it correctly," documented here.
 
 ## Erasure — what "delete" actually means
 
-CloudBitmaps gives you three erasure levers with different guarantees. Use them deliberately:
+CloudBitmaps gives you three levers with different guarantees. Use them deliberately:
 
 | Lever | API | Guarantee | Use for |
 |---|---|---|---|
-| **Logical remove** | `segment.remove(id)` / `removeMany` | *Immediate* — reads exclude the ID at once (tombstone; `effective = (Cold ∪ adds) \ removes`). The bit **physically persists** in the immutable Cold `.crbm` until that chunk is next compacted. | everyday "take this user out of this audience" |
-| **Physical purge** | `store.compact(ref, { owner })` on demand, or the compaction daemon (`compact-segments` / `runCompactionCycle`) on a schedule | A compaction folds the tombstone into a fresh generation and **physically drops** the bit. | meeting a physical-deletion deadline — *run compaction (the daemon on a schedule, or `store.compact` on demand) inside your SLA* |
-| **Crypto-shred** | `destroySegment` / `eraseNamespace` | *Instant + total* — destroys the segment's wrapped key, so **every** copy (current, prior generations, backups, WORM-locked objects) becomes unreadable without touching the bytes. Requires the segment to be encrypted. | whole-segment / tenant offboarding; erasure under immutable backups (see below) |
+| **Subject erasure** | `store.eraseSubject(id, { namespace })` | *Physical on return* — the segment's current generation is rewritten without the id (every chunk streamed through, one bit cleared), published **fenced on the generation it streamed**, and **the generation that held the bit is deleted before the call returns**. So is every other generation still holding the id: a retained *superseded* one (an ex-member dropped by a re-seed), and each one above the pointer after a `rollback` — generations a rollback can move the pointer onto again — while, when the current generation does not hold the id, the ones there that never held it stay as rollback targets (a rewrite is numbered above everything, so its `keep: 0` collection takes every older generation, above the pointer or below). Before it says `erased: true` the call lists the bucket and reads what is left, so **`erased: true` means no generation of the segment holds the id**, above the pointer or below it. `eraseSubject` reports a per-segment fault as an `error: …` ledger entry rather than throwing it, and a call that a racing writer overtook (a load, another erasure, or a rollback) as `erased: false, note: 'superseded'`, so "on return" is a claim about every segment whose entry says **`erased: true`** (a fault is an `error: …` entry, never `erased: true`). The store that performs the call stops serving the id immediately; **other processes converge on their own read TTL** — see [One process, and the rest of your fleet](#one-process-and-the-rest-of-your-fleet). Does **not** reach backups, replicas or noncurrent object versions — those hold the old generation until their own lifecycle removes it. | "forget this person" — GDPR Art. 17 |
+| **Dispose** | `store.dropSegment(ref, { confirmSegment })` | *Immediate* — the segment is tombstoned and its Storage generations deleted, reclaiming the space. **Check `generationsRemaining`:** if it is non-empty the space was *not* fully reclaimed and the drop should be re-run (a load that was already writing when the tombstone landed still finishes its object). Works on cleartext; on an encrypted segment it *also* crypto-shreds, on the terms in the next row. Does **not** reach noncurrent versions / replicas / PITR snapshots — deleting an object is weaker than removing a key. | retiring a dated bucket; rolling-window retention |
+| **Crypto-shred** | `destroySegment` / `eraseNamespace` | *At rest, complete once no copy of the wrapped key survives* — removes the segment's wrapped DEK(s) from its registry row in one compare-and-swap, without touching the bytes or any KEK. **Every** copy of the objects (current, prior generations, backups, WORM-locked objects) is then unreadable once no retained copy of that row still holds the wrapped key — noncurrent object versions, backups and PITR copies of the row included — or once every KEK that wrapped it is destroyed. Until then, a retained copy of the row and a KEK that wrapped it decrypt the segment, and a registry restore to a point before the shred makes it readable again; see *Erasure vs. backups / WORM* below, and [an optional registry expiry rule](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/disaster-recovery.md#optional-make-a-shred-durable-with-a-registry-expiry-rule) that makes a shred durable after a set number of days. Requires the segment to be encrypted. A process that had already opened the segment holds the **unwrapped** key in memory and keeps reading until it is told — call `store.invalidate(ref)` there; see [One process, and the rest of your fleet](#one-process-and-the-rest-of-your-fleet). | whole-segment / tenant offboarding; erasure under immutable backups (see below) |
 
 **Subject-wide erasure** (GDPR Art. 17 — "forget this person everywhere") is
-`store.eraseSubject(id, { owner, namespace })`:
-it removes the id from every **registered** segment it's in *and* force-compacts those segments on the spot, so
-the bit is physically gone from Cold on return — even for idle/archival segments organic compaction would never
-touch. It reuses the store's own drivers (so build the store with a raw cold driver + a registry). It returns an
-**erasure ledger** (per-segment: removed / physically-purged / which generation retired the bit) as your proof
-of deletion; persist it or route it to your audit sink. `store.subjectReport(id, { namespace })` answers the read side
-(Art. 15 — which segments an id is in). Run erasure without concurrently re-adding the same id; any ledger entry with
-`physicallyPurged:false` (a held compaction lease, or an isolated fault) needs a follow-up `store.compact(ref)` —
-re-running `eraseSubject` won't re-purge it.
+`store.eraseSubject(id, { namespace })` — or `{ allNamespaces: true }` to sweep every tenant deliberately. For
+every **registered** segment the id is in, it rewrites the current generation without the id, publishes the
+rewrite, and deletes every generation that held the bit — the one it replaced, a retained superseded one, and
+any above the pointer that a rollback left — so the bit is physically gone from the bucket on return, for idle
+and archival segments as much as busy ones. It checks rather than assumes: before an entry says `erased: true`,
+the call lists the segment's generations and reads each one it has not already seen without the id. There is no logical-then-physical gap and no
+scheduled step to wait for: an erasure *is* a new generation, the same shape as every other write in the library.
+It reuses the store's own drivers (so build the store with a backend). It returns an
+**erasure ledger** — one entry per segment the id was found in, `{ segment, namespace?, erased, fromGeneration,
+generation, note? }` — as your proof of deletion; persist it or route it to your audit sink, which also receives
+one `segment.rewrite { fromGeneration, generation }` event per rewrite when you pass `audit` — an id found only
+outside the current generation (a retained *superseded* one, or one above the pointer after a rollback) is
+collected rather than rewritten, so it emits no event and its ledger entry carries no `generation`. Segments the id is
+not in are not listed. `store.subjectReport(id, { namespace })` answers the read side (Art. 15 — which segments
+an id is in).
+
+Two rules. **Do not load a segment while erasing from it**: a load that lands after the rewrite carries whatever
+its source held, and the library cannot know that source was meant to exclude the id — fix the source first, or
+quiesce loads of the affected segments for the duration. A writer that lands *during* the rewrite is caught and
+the entry says `erased: false, note: 'superseded'` — because the publish was refused **by that fence**, because
+a racing **erasure** (which collects with `keep: 0`) deleted the generation the rewrite was still streaming, or
+because an operator's `rollback` moved the pointer while the call was deleting generations above it. Two
+erasures of different ids racing on one segment are safe: the loser's rewrite was derived from the generation the
+winner replaced, so it still holds the winner's id, and when it sits above the winner's pointer the loser deletes
+it before returning. **Read the ledger**: an `erased: false` entry means *this run* did not erase the id from that segment,
+which is not the same as the id still being present — if the racing writer was another erasure of the same id, it
+is already gone. For `'superseded'` re-run `eraseSubject`: it is idempotent, it erases the id if the id is still
+there, and a segment the id is no longer in is simply not listed — though "not listed" alone does not prove the
+id is gone, since a segment whose **registry row** has been purged is not scanned either and its objects outlive
+it as orphans (`store.checkConsistency()` does not report them: it visits registry rows only). Because a settled segment drops out of the
+ledger entirely, **the attestation for a subject is the ledger of the run that reported `erased: true`** — if you
+must hold one artifact per request, re-run until no entry carries a `'superseded'` note, and keep that run's
+ledger alongside any earlier one. An `error: …` note is an isolated per-segment fault, and it is reported rather
+than swallowed because it can be the signal that a *published* rewrite's physical half did not complete **on that
+call**. If it occurred *after* the rewrite was published — a Storage `delete` fault, a collection pass that
+could not prove the segment was still the same one, or a generation still holding the id when the bucket is listed at the end, such as one an operator
+rolled the pointer onto while the rewrite was collecting — then the pointer has already moved, and the re-run
+searches every generation in the bucket, not only the current one. A transient fault on the publish itself is an `error: …` note too, and there the rewrite may or
+may not have become current: the re-run settles it either way, rewriting the id out if the pointer did not move and
+finding it in the generation the rewrite replaced if it did. (The collection-side errors just listed — a Storage `delete` fault, a pass that could not prove the segment was the
+same one, a generation still holding the id — can also come from the collect-only path, where the bit was found
+only outside the current generation and nothing was published; there the pointer has not moved, and a re-run
+simply repeats the attempt.) **Read what the re-run says.** Usually it reports `erased: true` against the
+generation it found the id in, which is the receipt the failed call could not give you. If a racing collector took
+that generation first it reports nothing for the segment: the bit is gone, but no run holds a receipt for it, so
+keep the failed call's error alongside your ledger. And if the segment's registry row has since been purged, it is
+not a segment any more and `eraseSubject` does not scan it at all — anything left in its bucket is an **orphan**,
+which neither `store.checkConsistency()` (it visits registry rows only) nor the collection a load and the retention
+sweep run (it collects nothing without a row) reaches. List those objects with `store.generations(ref)` and delete them with
+`store.dropSegment(ref, { confirmSegment: ref.segment })`, which leaves a tombstone row fencing the name. An empty
+ledger is not by itself proof the id is gone.
+
+**Do not roll a segment back while erasing from it.** A rollback that lands mid-erasure can move the pointer onto a
+generation the erasure did not rewrite, so the id can be current again. Roll back before the erasure starts or
+after it returns, and re-run any erasure a rollback interrupted.
+
+### One process, and the rest of your fleet
+
+Erasure and crypto-shred are **immediate in storage and immediate in the store whose verb performed them**. They
+are not immediate in any *other* store, in the same process or another, and this library ships nothing that could
+make them so — there is no daemon, no bus, and no connection between two stores that happen to point at the same
+bucket.
+
+| | when the id stops being readable |
+|---|---|
+| storage | on return — the generation holding it is deleted, or the wrapped DEK is removed from the segment's current registry row (a crypto-shred is complete once no retained copy of that row holds it, or every KEK that wrapped it is destroyed; see *Erasure vs. backups / WORM* below) |
+| the store whose verb made the call (`eraseSubject`, `dropSegment`, `retireExpired`) | on return — it invalidates what it cached, and its pins then fail |
+| another store, with a registry and a `cache.genTtlMs` above 0 | within `cache.genTtlMs` (default 2 s), when its snapshot re-resolves, **while the registry can be read** (see below) |
+| another store with **no registry** (a bare `IStorageDriver`), with `cache: { genTtlMs: 0 }`, or on a storage source built with **no clock** | **no bound** — only when its caches happen to let the segment go, or something tells it |
+| a pinned handle (`seg.pin()`) in another store | **no bound** — until that store's reader cache evicts the pin's reader and its chunk cache evicts the chunks the pin decoded, or something tells it |
+
+**An outage of the registry extends that bound.** A refresh that cannot read the row because of a transient fault
+(throttling, a 5xx, a dropped connection) keeps serving the generation the reader already holds, and the key it
+unwrapped, and asks again 500 ms later, or after `cache.genTtlMs` if that is shorter. So while the registry cannot
+be read, a shred or a drop is not seen, and the store converges within one retry of the registry answering. Any
+other failure of the refresh (an access denial, a row that will not parse) is not ridden out: the read that finds it
+fails with that error, and the reader is dropped with the key it held.
+
+`destroySegment` and `eraseNamespace` are free functions over raw drivers, not verbs of a store, so for them every
+store is another store, one in the same process included.
+
+The last two rows are the ones to design around. `cache: { genTtlMs: 0 }` turns the timed refresh off, and is a
+legitimate setting for a read-only replica of immutable data — but a store set that way has no bound on when it
+observes a shred: until its reader cache lets the segment go, it keeps decrypting with the key it already unwrapped.
+A pinned handle is that case in any store, clock or none, and holds on longer: its reader keeps the key until the
+reader cache evicts it, and after that the pin still answers `has()` for the ids in chunks it decoded, until the
+chunk cache drops them. If a
+compliance deadline depends on every reader converging, fan the reference out to your fleet and have each
+process call `store.invalidate(ref)`; that is the hook, and delivering it is yours because the transport is
+yours.
+
+A read served from a stale cache is bounded by the same window and answers from memory, so nothing on the
+storage side — a bucket policy, a lifecycle rule, the object's own deletion — can shorten it.
 
 **Your exit path** (and a building block for a **data-portability / Art. 20** response): `store.exportSegments(sink,
-{ format })` (and the `export-segments` CLI) dumps every registered segment's effective set to a portable file —
-`roaring` (loadable by any roaring library) or `ndjson` (newline ids) — readable without CloudBitmaps. It's a
-**controller-side bulk dump** (all segments, opaque ids), not a per-subject deliverable — the per-subject rights
-are `subjectReport` (Art. 15) / `eraseSubject` (Art. 17). Encrypted segments are decrypted transparently if the
-store has the keystore, so the **export is cleartext — protect it** (the CLI writes owner-only files; also encrypt
-the dump at rest, restrict access, and delete it when done).
-
-**Honest limits.** Logical `remove()` is *not* physical deletion on its own — you need compaction (which
-`eraseSubject` forces, or a scheduled daemon), or crypto-shred. **Per-subject crypto-shred is infeasible** (a
-subject's bit is co-mingled with millions of others in one shared container), so single-subject erasure goes
-through remove + compaction (`eraseSubject`), while crypto-shred (`destroySegment`/`eraseNamespace`) cleanly
-handles segment/tenant-level erasure and is the only erasure that survives immutable backups / WORM.
-
-**Erasure vs. backups / WORM (the trap).** If you enable S3 versioning, Object Lock, or DynamoDB PITR for
-durability, a `remove()`+compaction *does not* reach the retained copies — the deleted bit survives in
-noncurrent versions, locked objects, and backups. **Crypto-shred is the only erasure that survives all of
-them**, because it destroys the key, not the bytes. So: use **per-segment/tenant encryption** as your erasure
-posture under immutable storage; reserve S3 Object Lock **COMPLIANCE** mode for data under a genuine legal
+{ format })` (and the `export-segments` CLI) dumps every registered segment's current generation to a portable
+file — `roaring` (loadable by any roaring library) or `ndjson` (newline ids) — readable without CloudBitmaps.
+It's a **controller-side bulk dump** (all segments, opaque ids), not a per-subject deliverable — the per-subject
+rights are `subjectReport` (Art. 15) / `eraseSubject` (Art. 17). Encrypted segments are decrypted transparently
+if the store has the keystore, so the **export is cleartext — protect it** (the CLI writes owner-only files;
+also encrypt the dump at rest, restrict access, and delete it when done).
+
+**Honest limits.** Subject erasure deletes the object that held the bit; it does not reach the copies of that
+object your storage keeps on its own — noncurrent versions (S3 versioning), cross-region replicas, backups and
+WORM-locked copies hold the old generation until their own lifecycle expires it. **Per-subject crypto-shred is
+infeasible** (one DEK covers the whole segment, and a subject's bit is co-mingled with millions of others), so
+single-subject erasure is a rewrite (`eraseSubject`), while crypto-shred (`destroySegment` / `eraseNamespace`)
+handles segment/tenant-level erasure and is the only erasure that reaches immutable backups / WORM copies of the
+objects — once no copy of the segment's registry row still holds its wrapped key, or every KEK that wrapped it
+is destroyed (below). And a
+**materialized segment** (`intersectInto` / `unionInto` / `andNotInto`) is a point-in-time snapshot of its
+inputs: erasing a subject from a source does not touch a destination computed earlier — which is exactly why
+`eraseSubject` scans *every* registered segment, destinations included, rather than erasing per source. What that
+costs is one unit of the per-op `budget` for each segment, and one more for each generation it opens in a segment
+whose current generation lacks the id (it searches every generation left in the bucket, since a retained one can
+still hold the bit). A segment that runs the budget out is listed `erased: false` with an `error:` note, with
+nothing of it deleted before the refusal (a segment that holds the id is rewritten first, and the last check
+after it can be the one refused, which the same note reports); re-run with a higher `budget`.
+
+**Erasure vs. backups / WORM (the trap).** If you enable S3 versioning, Object Lock, or registry PITR for
+durability, a subject-erasure rewrite *does not* reach the retained copies — the deleted bit survives in
+noncurrent versions, locked objects, and backups, and a restore that brings one of them back brings the erased id
+back with it (the [disaster-recovery guide](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/disaster-recovery.md#restore-procedure) re-runs the
+erasures made after the restore point). **Crypto-shred reaches the retained copies of the objects without touching
+them**, because it removes the key, not the bytes — but it removes the key from one place: the segment's current
+registry row. The same versioning, PITR and backups keep earlier copies of that row, and every copy written since
+the segment's first encrypted load still holds the wrapped key. **A shred is complete once no retained copy of the
+registry row still holds the wrapped key** (noncurrent object versions, backups and PITR copies included), **or
+once every KEK that wrapped it is destroyed.** Until then, a retained copy of the row plus a KEK that wrapped it
+decrypts the segment, and a registry restore to a point before the shred makes it readable again, which is why the
+disaster-recovery guide re-shreds after a restore. Versioning on the registry prefix is a requirement of that
+guide's restore procedure, so with it on, when a shred completes is set by how long your storage keeps the row's
+noncurrent versions and backups — the same history a registry restore picks from — and an Object Lock default
+retention on a bucket that also holds the registry locks those versions for that retention period. Destroying a
+KEK happens in your keystore, not through the library, and a segment is shredded that way only once every KEK in
+its wrappings is destroyed: with `InProcessKeystore`, the KEK that was active when the segment was first loaded
+and, if one was configured, the recovery KEK, which wraps every segment's DEK a second time. Destroying those
+shreds every segment they wrapped. So: use **per-segment/tenant encryption** as your erasure posture under
+immutable objects, and count the registry row's retained copies in when you state that a shred is complete; give
+noncurrent versions of the objects a short expiry (a lifecycle rule on *noncurrent* versions is fine — it is
+*current* generations that a rule must never expire, see below — though it also bounds how far back the
+disaster-recovery guide can restore); reserve S3 Object Lock **COMPLIANCE** mode for data under a genuine legal
 hold (it *cannot* be deleted before its retention date, by anyone — incompatible with on-demand erasure), and
 prefer **GOVERNANCE** mode where erasure must remain possible.
 
 ## Retention & data minimization
 
-The library does not age data out for you — segments grow until you prune them, and "cheap to keep forever" is
-a storage-limitation anti-pattern if the data is personal. **Retention policy is yours.** Practical patterns:
-
-- **Rolling windows** (e.g. "active this week"): keep N daily sub-segments, union them for reads, and drop the
-  oldest wholesale — turning retention into cheap whole-segment deletion (an object delete or crypto-shred),
-  not per-bit aging.
-- **Scheduled compaction** enforces any tombstones you've written for aged-out members within your window.
-- Surface segment age/size via the **metrics sink** so unbounded growth is visible, not silent.
+The library will age a **segment** out for you once you tell it when to — but it will not schedule that for you,
+and it will never age out an individual **id** (a bitmap stores ids, not timestamps). "Cheap to keep forever" is a
+storage-limitation anti-pattern if the data is personal, so: **the policy is yours to set, the heartbeat is yours
+to run, and the deletion is ours to perform correctly.** Practical patterns:
+
+- **A per-segment expiry** — `store.setRetention(ref, { expiresAt })` records when a segment becomes eligible for
+  retirement, and `store.retireExpired()` retires every expired segment (through `dropSegment`, so the ordering
+  and the storage reclamation below are inherited rather than reimplemented). The expiry is an absolute instant
+  the writer sets; the sweep is a call **you** schedule (EventBridge, a CronJob, cron, a queue job), because this
+  library starts no background timer. Bounded per cycle, previewable with `dryRun`, and it returns a per-segment
+  ledger you should inspect — an entry can say a segment's storage was not fully reclaimed.
+- **Rolling windows** (e.g. "active this week"): keep N daily sub-segments, union them for reads, and let the
+  policy above retire the oldest wholesale — turning retention into cheap whole-segment disposal, not per-bit aging.
+- **A reload is the per-id retention.** Since a segment's content is whatever its last load said, ageing members
+  out *within* a segment is a matter of loading the next generation from a source that no longer includes them.
+  The registry row's `retention` is untouched by a load, an `*Into` materialization or an erasure rewrite, so the
+  `expiresAt` you set stays put across all of them.
+- Surface segment age and size so unbounded growth is visible, not silent. The **metrics sink** carries neither (its `storage.get` events count the bytes of each read, not a segment's size).
+  `seg.count()` gives a segment's cardinality; `seg.costReport()` prices its current generation's measured size
+  (`monthlyUSD.byOp.storage`, $0 when the storage source cannot measure it), and does not count superseded
+  generations still in the bucket, which `store.generations(ref)` lists; and its registry row carries `createdAt`
+  and `updatedAt`, which the backend's `registry.list()` returns for every row.
+
+**Be precise about what "drop the oldest" involves**, because the three levers differ in what they guarantee:
+
+- **`store.dropSegment(ref, { confirmSegment })` is the retention primitive**, and `setRetention` +
+  `retireExpired` is the policy-driven form that calls it — and its result must be inspected, not
+  assumed: `generationsRemaining` non-empty means bytes survived and the call should be repeated. It tombstones
+  the segment and deletes **every Storage generation** — so the space is actually reclaimed. It works on a
+  cleartext segment, and on an encrypted one it *also* crypto-shreds, making it a strict superset there.
+- **`destroySegment` crypto-shreds** — it removes the wrapped key from the registry row, so the Storage bytes
+  become unreadable *everywhere including backups, replicas and WORM-locked copies* once no retained copy of that
+  row still holds the key, or every KEK that wrapped it is destroyed (see the trap above), which no object
+  deletion can achieve. But **the objects
+  remain in your bucket** and you keep paying for them, and it **requires encryption at rest** (a cleartext
+  segment has no key to discard; `allowCleartext` writes the tombstone while leaving the Storage bytes readable — and
+  still in the bucket).
+- **The collection a `load` runs** (its `keep` window; an `*Into` runs it when given `keep`) deletes only
+  *superseded* generations, never the current one — with `keep: 0` it is what a subject erasure runs to remove the
+  generations below the pointer that held the bit. A holder *above* the pointer
+  is outside its range, so the erasure deletes that one itself.
+
+So the two are complements, not alternatives: **`dropSegment` for "stop paying for it", `destroySegment` for
+"it must be unreadable even in backups"** — and on an encrypted segment `dropSegment` gives you both at once.
+
+> ⚠️ **Do not use an object-store lifecycle rule as your retention mechanism.** It deletes the bytes while the
+> registry still points at them, which is the `missing-storage-generation` torn state — and it surfaces
+> *intermittently*, because a read consults the in-process cache before Storage, so it passes testing on a warm
+> process and starts failing after a restart. A lifecycle rule is a fine **backstop** for orphans left by a
+> failed `dropSegment`, and a fine way to expire *noncurrent* object versions; give it a window comfortably
+> longer than your retention and never let it touch a current generation.
+
+> ⚠️ **Never let a lifecycle rule expire a registry row's current version or its tombstone** — a separate trap,
+> and a worse one. Deleting a registry row does not remove it: it writes a **tombstone** carrying a counter that only
+> ever advances, which is what makes the row's token unique for all time. Expire those tombstones and a segment
+> re-created under the same name starts the counter again and re-issues a token that was already used. That
+> token is the segment's **identity**: it is what a cached reader, a fenced publish and a generation-collection
+> pass each compare to decide whether two observations describe the same segment. Re-issue one and they can all
+> answer "yes" about a segment that no longer exists — serving a deleted incarnation's data, or collecting a
+> live one's objects. Storage for a tombstone is a few bytes per retired segment; treat it as permanent. A rule
+> on *noncurrent* versions of the registry prefix touches neither the current row nor its tombstone; what it
+> shortens is the history a registry restore picks from, and the time a crypto-shred takes to complete (above).
+
+Full detail, including the dated-bucket pattern and the pitfalls, is in the retention section of the
+[retention guide](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/retention.md).
 
 ## Legal hold
 
 A litigation/regulatory hold *forbids* deletion — the opposite of erasure — and can apply to the same segment.
 **Enforce a hold with S3 Object Lock**, which is the real, tamper-proof mechanism: in **COMPLIANCE** mode a
-locked Cold object *cannot* be deleted before its retention date by anyone (not even the account root); in
+locked Storage object *cannot* be deleted before its retention date by anyone (not even the account root); in
 **GOVERNANCE** mode a privileged role can override. To place a hold on a segment:
 
-1. Enable **Object Lock** on the segment's Cold `.crbm` objects (and enable versioning) for the hold period.
-2. **Exclude the segment from your compaction and erasure runs** — don't call `eraseSubject`, `destroySegment`,
-   or the compaction daemon against held segments, so the current generation (and its members) is preserved.
+1. Enable **Object Lock** on the segment's Storage `.crbm` objects (and enable versioning) for the hold period.
+2. **Exclude the segment from your erasure runs and your loads** — don't call `eraseSubject` in a scope that
+   reaches it, `destroySegment`, **`dropSegment`**, or load a new generation onto a held segment, so the current
+   generation (and its members) is preserved. Because `eraseSubject` scans a whole namespace, the practical
+   shape is to keep held segments in a namespace of their own. **`dropSegment` matters most here**: it is the only
+   call that deletes the bytes a hold exists to protect. In COMPLIANCE mode Object Lock will refuse the delete,
+   so the drop reports the generation as not deleted rather than silently succeeding — but the tombstone is
+   still written, so the segment reads as empty while the locked bytes remain. **Do not rely on Object Lock to be
+   the guard**, and note how exclusion works today: `retireExpired` has **no exclusion predicate**, so a held
+   segment must either *never carry a retention policy*, or have it removed with `clearRetention` for the
+   duration of the hold and re-set afterwards. Recording the hold itself is free — any key in the row's
+   `retention` metadata other than `expiresAt` and `retiredBySweepAt` (the stamp the sweep puts on its own
+   retirements, which marks a tombstone as one it may purge) is yours and survives both `setRetention` and
+   `clearRetention`, so `{ legalHold: 'case-1234' }` is a durable marker; it is simply not yet something the
+   sweep reads. `setRetention` writes only `expiresAt`, so write the marker yourself, with `compareAndSwap` on the
+   backend's `registry`. (An `exclude` predicate is under consideration; say so on an issue if you need it.)
 3. Decide hold-vs-erasure precedence when both apply to the same subject — that is a **legal determination**;
    under a hold, erasure is suspended.
 
 CloudBitmaps deliberately does **not** ship a native `legalHold` flag: Object Lock is a stronger guarantee than
-an in-library flag (which our own daemon could respect but a direct caller could bypass), so a flag would be
-advisory where Object Lock is enforced at rest. If a real deployment needs a library-managed hold that the
-daemon refuses to purge, it's a clean fast-follow — but the enforced posture is Object Lock + operational
-exclusion, documented here (P10).
+an in-library flag (which our own erasure and sweep could respect but a direct caller could bypass), so a flag
+would be advisory where Object Lock is enforced at rest. If a real deployment needs a library-managed hold that
+`eraseSubject` and `retireExpired` refuse to touch, it's a clean fast-follow — but the enforced posture is
+Object Lock + operational exclusion, documented in this section.
 
 ## Audit & accountability (GDPR Art. 30 / Art. 5(2))
 
 Wire the **audit sink** (`IAuditSink`) to get an append-only, vendor-neutral record of the compliance-relevant
-state changes — `segment.publish`, `segment.compact`, `segment.erase` (a genuine crypto-shred), and
-`namespace.erase` — for your audit log / SIEM. It is off by default and exception-safe. See the
-[dashboards guide](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/dashboards.md). An erasure *ledger* — per-subject-request proof of physical
-deletion — is returned by **`eraseSubject`** (shipped in Phase 6b). `subjectReport` is the read side and
-returns only which segments an id is in; it performs no deletion and issues no ledger.
+state changes — `segment.publish` (a loaded generation became current), `segment.load-refused` (a load that did
+not publish, because its guard refused it or another writer got there first), `segment.rollback` (an operator moved
+the pointer to a generation it named), `segment.rewrite` (a subject-erasure
+rewrite: `fromGeneration` → `generation`), `segment.erase` (a genuine crypto-shred), `segment.dispose` (a
+`dropSegment`, including every retirement the sweep performs) and `namespace.erase` — for your audit log / SIEM.
+It is off by default and exception-safe. See the [dashboards guide](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/dashboards.md), which also says
+which event is the receipt for which claim. An erasure *ledger* — per-subject-request proof of physical deletion
+— is returned by **`eraseSubject`**. `subjectReport` is the read side and returns only which segments an id is
+in; it performs no deletion and issues no ledger.
 
 **Log hygiene:** `segment` / `namespace` are *your* strings and may encode sensitive purpose; IDs are personal
 data. The library never logs bitmap contents or raw IDs, but **you** should treat segment names and IDs as PII
 in your own logs, error reporting, and metric/trace tags — hash or redact them, and prefer opaque/coded
-segment names for sensitive segments (keeping the human label in your own classified registry).
+segment names for sensitive segments (keeping the human label in your own classified registry). The name rules are deliberately permissive — any non-empty string — so a name can carry an email
+address, free text, or anything else an upstream system hands you. That makes this advice stronger rather than
+weaker: the library will store what you name, and will not sanitise it for you.
 
 ## Shared-responsibility matrix
 
@@ -144,12 +336,12 @@ segment names for sensitive segments (keeping the human label in your own classi
 |---|---|---|
 | **Controller/processor role** | an embedded library; sends nothing to us | be the controller/processor; run your own DPAs with *your* cloud providers |
 | **Encryption at rest** | AES-256-GCM envelope encryption, BYOK keystore (`InProcessKeystore`), per-segment DEK + active/recovery KEK | hold and protect your keys (KMS/HSM); enable encryption for sensitive segments |
-| **Erasure** | `remove` (logical), compaction (physical), `destroySegment`/`eraseNamespace` (crypto-shred) | run compaction within your SLA; choose crypto-shred under WORM/backups; classify what needs erasing |
-| **Residency** | region-agnostic drivers; you choose every location | wire region-correct drivers; run compute in-region; assess transfers |
+| **Erasure** | `eraseSubject` (per-id rewrite, physical on return), `dropSegment` (dispose), `destroySegment`/`eraseNamespace` (crypto-shred) | read the erasure ledger and re-run on `erased: false`; don't load a segment while erasing from it; choose crypto-shred under WORM/backups, and count the registry row's retained copies in when a shred is complete; re-apply erasures after a restore; classify what needs erasing |
+| **Residency** | region-agnostic drivers; you choose every location | wire region-correct drivers; run loaders and compute in-region; assess transfers |
 | **Classification** | opaque handling; a place to keep sensitive segments encrypted + audited | classify your segments (the library can't infer sensitivity) |
-| **Retention** | scheduled compaction; rolling-window pattern | set and enforce retention windows |
-| **Legal hold** | Object Lock guidance (the enforced mechanism; no in-library flag) | place holds via Object Lock; exclude held segments from compaction/erasure; decide precedence |
-| **Accountability** | `IAuditSink` (publish/compact/erase events) | route it to durable, append-only storage; keep your Art. 30 record |
+| **Retention** | per-segment `expiresAt` + `retireExpired` (bounded, previewable, ledgered); rolling-window pattern | set the policy per segment; **schedule the sweep yourself** and alarm on its ledger |
+| **Legal hold** | Object Lock guidance (the enforced mechanism; no in-library flag, and no sweep-level exclusion — see above) | place holds via Object Lock; keep a held segment out of the retention sweep by not giving it a policy, and out of `eraseSubject`'s scope; decide precedence |
+| **Accountability** | `IAuditSink` (publish / rewrite / erase / dispose events) | route it to durable, append-only storage; keep your Art. 30 record |
 | **Telemetry** | **none** — no phone-home | (nothing — there is nothing to disable) |
 
 ## DPIA skeleton (for deployments with sensitive segments)
@@ -159,13 +351,13 @@ A minimal Data-Protection-Impact-Assessment outline to adapt:
 1. **Processing description** — which segments, what each membership *means*, source of the IDs, volume.
 2. **Necessity & proportionality** — lawful basis per segment (esp. Art. 9 special-category); why membership
    is retained and for how long.
-3. **Data flow & residency** — Warm/Cold regions, where compute (HOT cache, intersection) runs, cross-border
-   transfers and their safeguards.
+3. **Data flow & residency** — Storage and registry regions, where compute (loaders, the cache, intersection)
+   runs, cross-border transfers and their safeguards.
 4. **Risks** — re-identification, sensitive inference (incl. *derived* segments from intersections — treat a
    `paying ∩ pregnant` result as at least as sensitive as its inputs, and note it's a point-in-time snapshot
    that does not auto-honour later erasures of source members), breach scope (incl. process RAM / dumps).
-5. **Controls** — encryption + crypto-shred, erasure SLA (scheduled compaction), retention windows, audit
-   sink, access control on keys and storage, redaction of names/IDs in logs.
+5. **Controls** — encryption + crypto-shred, subject erasure that is physical on return (`eraseSubject`),
+   retention windows, audit sink, access control on keys and storage, redaction of names/IDs in logs.
 6. **Residual risk & sign-off** — DPO review.
 
 ## Art. 30 record — mapping template
@@ -174,14 +366,14 @@ Map CloudBitmaps' processing onto the categories a record of processing needs:
 
 | Art. 30 field | CloudBitmaps mapping |
 |---|---|
-| Categories of processing | storage of set-membership; set intersection; compaction; caching |
+| Categories of processing | storage of set-membership; set intersection (incl. materialized results); loading of generations; caching |
 | Categories of data subjects / data | your IDs' subjects; membership (possibly special-category) |
 | Recipients | none external to your infrastructure (no sub-processor) |
 | Transfers | any cross-region driver/compute topology *you* configure |
-| Retention | your per-segment policy (enforced via scheduled compaction / rolling windows) |
+| Retention | your per-segment policy — `setRetention(ref, { expiresAt })`, enforced by a `retireExpired` sweep **you schedule** |
 | Security measures | AES-256-GCM at rest, crypto-shred erasure, audit sink, your access controls |
 
 ---
 
-**See also:** the [getting-started guide](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/getting-started.md) (§9 encryption/crypto-shred, §10
-metrics, §12 audit) and the [dashboards guide](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/dashboards.md).
+**See also:** the [encryption](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/encryption.md), [observability](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/observability.md) (metrics and
+audit) and [erasure](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/erasure.md) guides, and the [dashboards guide](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/dashboards.md).