@cloudbitmaps/roaring 0.20.0 → 0.21.0

package/PRIVACY.md

@@ -43,7 +43,7 @@ from constructing a cross-region topology. The points where personal data moves
 | Location | What's there | Residency note |
 |---|---|---|
 | **Storage** (object store) | immutable `.crbm` generations — every generation a segment has had, until a superseded one is collected. Azure Blob objects, and GCS objects above the simple-upload threshold, also carry a random write id in their metadata (`cbwid`): 128 random bits, no data from the bitmap or the source | the region of the bucket you wire |
-| **Registry** (S3 / GCS / Azure Blob / local) | one row per segment: the current-generation pointer, wrapped keys, retention metadata, and the current generation's id count and your metadata (sealed when the segment is encrypted) — no IDs, unless you put one in the metadata, which you must not. An Azure Blob row also carries the random write id in its metadata | the region of the bucket you wire |
+| **Registry** (S3 / GCS / Azure Blob / local) | one row per segment: its namespace and name, its status and timestamps, the current-generation pointer, wrapped keys and any key id, its `retention` and `residency` metadata, the current generation's id count, its object's size and footer checksum and your metadata (those three sealed when the segment is encrypted), the generations its loads keep, any leases, and a token and `pointerId` — no IDs, unless you put one in the metadata or a name, which you must not. An Azure Blob row also carries the random write id in its metadata | the region of the bucket you wire |
 | **cache** (process RAM) | decoded chunks, and the stored chunk bytes (ciphertext on an encrypted segment) a reader keeps of a small generation it read whole; bounded LRU | **wherever your process/Lambda runs** — an EU segment queried from a US function is processed in the US |
 | **Loads and rewrites** (`store.load()`, the `*Into` verbs, `store.materializeMany()`, `eraseSubject`) | read your source (or existing generations), write a new generation; a fed operand of `materializeMany` is read from your own process's memory | run wherever you run them — a loader in one region writing to a bucket in another is a transfer |
 | **Intersection** | pulls chunks from N segments into one process | co-locates those segments in one region |
@@ -88,15 +88,30 @@ no read of it can open, is not listed either: the audit sink gets `segment.colle
 the read side (Art. 15 — which segments
 an id is in).
 
-Two rules. **Do not load a segment while erasing from it**: a load that lands after the rewrite carries whatever
+An erasure that finds the id in no generation but an older one also writes the segment's registry row once, before it
+deletes anything, when the row's ledger of known generations names a generation the call is about to delete: it takes
+those names out of the ledger first (see [how it stays correct](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/erasure.md#how-it-stays-correct)). That write
+is a timing signal. Someone who can read the registry and knows when an erasure ran can tell that this segment's row
+changed then, and so that the segment held the subject in a generation older than its current one. The row also
+changes at erasure time when the segment's current generation held the id, or one above the pointer, a first load's
+object on no pointer, or an object no read could open. The entries the write removes from the ledger show which
+generations went, and for a holder above the pointer that is exactly the holders; the ledger names no id and no size,
+and is written in the clear on every row, an encrypted one included, carrying a generation's number and its object's
+footer checksum. When a load starts a ledger while the call runs, or a rollback or a load makes the row name a holder
+again, the row changes a second time.
+
+Two of the rules while you erase. **Do not load a segment while erasing from it**: a load that lands after the rewrite carries whatever
 its source held, and the library cannot know that source was meant to exclude the id — fix the source first, or
 quiesce loads of the affected segments for the duration. A writer that lands *during* the rewrite is caught and
 the entry says `erased: false, note: 'superseded'` — because the publish was refused **by that fence**, because
 a racing **erasure** (which collects with `keep: 0`) deleted the generation the rewrite was still streaming, or
-because an operator's `rollback` moved the pointer while the call was deleting generations above it. Two
-erasures of different ids racing on one segment are safe: the loser's rewrite was derived from the generation the
-winner replaced, so it still holds the winner's id, and when it sits above the winner's pointer the loser deletes
-it before returning. **Read the ledger**: an `erased: false` entry means *this run* did not erase the id from that segment,
+because an operator's `rollback` moved the pointer while the call was deleting generations above it. When two
+erasures of different ids race on one segment, the loser's rewrite was derived from the generation the winner
+replaced, so it still holds the winner's id, and when it sits above the winner's pointer the loser deletes it before
+returning, unless writes other than lease writes and other erasures' renewals beat its renewal of the segment's
+row 8 times: it then keeps that object, one `rollback({ allowUnknown: true })` from being served, until a later call
+removes it: an erasure that rewrites the segment (the loser's re-run, while the loser's id is still in the current
+generation), an erasure of the winner's id, or the collection of loads that number past it ([how it stays correct](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/erasure.md#how-it-stays-correct)). **Read the ledger**: an `erased: false` entry means *this run* did not erase the id from that segment,
 which is not the same as the id still being present — if the racing writer was another erasure of the same id, it
 is already gone. For `'superseded'` re-run `eraseSubject`: it is idempotent, it erases the id if the id is still
 there, and a segment the id is no longer in is simply not listed — though "not listed" alone does not prove the
@@ -134,7 +149,7 @@ one that does not, it can, and the row then names a missing object, which `store
 ([a number taken again](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/erasure.md#how-it-stays-correct)). On an encrypted store such an object cannot
 be searched until its load publishes the key, so it is deleted whatever id is erased: an erasure of any id refuses an
 encrypted first load in flight that has written its object onto a segment whose row was made ahead of its data, and
-that load is re-run ([the erasure guide](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/erasure.md#two-rules-while-you-erase)). On an encrypted segment,
+that load is re-run ([the erasure guide](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/erasure.md#three-rules-while-you-erase)). On an encrypted segment,
 an object sealed under a key its row does not hold, and on a cleartext segment with a generation, an encrypted object,
 open for no read of the segment (the object of a first load that lost the race to the one that published, or
 crashed): each is deleted the same way, whatever id is erased, wherever an erasure meets it, except as the current
@@ -156,7 +171,7 @@ A call with a feed reads operands the library never stores, which no erasure can
 Erasure and crypto-shred are **immediate in storage and immediate in the store whose verb performed them**, for
 every read that starts after the verb returns. A read already in progress in that store can still yield an erased id
 from a chunk it had requested before the verb ran: up to 32 chunks for `iterate` and `count`, and up to `concurrency` + 1
-keys (33 by default) for a combine, the key it is handing out and the `concurrency` keys it had already requested. A combine or `iterate` reads ranges of the object, and resolves the segment again before it serves each chunk, one held in its cache included, as a read of that chunk alone does, so the ranges it had already requested are dropped, not served. They are not immediate in any *other* store, in the same process or another, and this library ships nothing that could
+keys (33 by default) for a combine. That is the bound: the chunk or key a read is handing out and what it had already requested, which a `count` that reads chunks (on a source with no index of its own) reaches. A running `iterate` or combine on the library's own sources stops sooner: the verb invalidates the segment, so before the next key it takes the read lists its chunk keys again from the generation current then and drops what it had requested, and so yields the id only from the chunk or key it is handing out. A combine or `iterate` reads ranges of the object, and resolves the segment again before it serves each chunk, one held in its cache included, as a read of that chunk alone does, so the ranges it had already requested are dropped, not served. A pinned read in progress does not move to the rewritten generation: before its next chunk it opens its object again, and fails with `NotFoundError` once the erasure has deleted it, so it yields no more than that bound either. They are not immediate in any *other* store, in the same process or another, and this library ships nothing that could
 make them so — there is no daemon, no bus, and no connection between two stores that happen to point at the same
 bucket.
 
@@ -166,7 +181,7 @@ bucket.
 | the store whose verb made the call (`eraseSubject`, `dropSegment`, `retireExpired`) | on return, for every read that starts after it — it invalidates what it cached, and its pins then fail; a read already in progress there can still yield it from a chunk it had requested before |
 | another store, with a registry and a `cache.genTtlMs` above 0 | within `cache.genTtlMs` (default 2 s), when its snapshot re-resolves, **while the registry can be read** (see below) |
 | another store with **no registry** (a bare `IStorageDriver`), with `cache: { genTtlMs: 0 }`, or on a storage source built with **no clock** | **no bound** — only when its caches happen to let the segment go, or something tells it |
-| `seg.pinAt` of a generation the erasure rewrote, in any store | on return it fails with `NotFoundError`, since the object is deleted (and, with a registry, the row has moved on); during the erasure, between the rewrite's publish and its delete of the old generation, it can still open that generation, and a handle opened then is a pin, covered by the row below |
+| `seg.pinAt` of a generation the erasure rewrote, in any store | on return it fails with `NotFoundError`, since the object is deleted (and, with a registry, the row has moved on); during the erasure, between the rewrite's publish and its delete of the old generation, it can still open that generation, and a handle opened then is a pin, covered by the rows for the store whose verb made the call and for a pinned handle in another store |
 | a `materializeMany` call with a feed, and an erasure by any path but this store's `eraseSubject` | **no bound**: the call is refused only by this store's `eraseSubject` |
 | a pinned handle (`seg.pin()`) in another store | **no bound** — until that store's reader cache evicts the pin's reader (in a store with a timed pointer refresh, a small generation's reader holds all of its chunks, decoded or not) and its chunk cache evicts the chunks the pin decoded, or something tells it |
 | a leased pinned handle (`seg.pin({ leaseUntil })`) in another store | until its lease ends, at most 14 days after it was taken: its reads then throw `LeaseExpiredError`, whether or not its reader still holds the chunks. Until then, as for any pinned handle in the row above |
@@ -213,8 +228,9 @@ leased handle stops answering when its lease ends, and its reads then throw `Lea
 registry row is a random holder id, a generation number and an instant, with no id and no count.
 
 **Your exit path** (and a building block for a **data-portability / Art. 20** response): `store.exportSegments(sink,
-{ format })` (and the `export-segments` CLI) dumps every registered segment's current generation to a portable
-file — `roaring` (loadable by any roaring library) or `ndjson` (newline ids) — readable without CloudBitmaps.
+{ format })` (and the `export-segments` CLI) dumps the current generation of every registered segment that is not
+destroyed (crypto-shredded or dropped) to a portable
+file — `roaring` (loadable by any roaring library) or `ndjson` (newline ids) — readable without CloudBitmaps. A destroyed segment resolves no generation, so it is not exported: the manifest's `skipped` lists it, and `segments`, `failed` and `skipped` together account for every segment row the registry listed.
 It's a **controller-side bulk dump** (all segments, opaque ids), not a per-subject deliverable — the per-subject
 rights are `subjectReport` (Art. 15) / `eraseSubject` (Art. 17). Encrypted segments are decrypted transparently
 if the store has the keystore, so the **export is cleartext — protect it** (the CLI writes owner-only files;

package/README.md

@@ -78,7 +78,7 @@ name.
 A cold `intersect` of two 500,000-id segments sharing 100 of their 1,999 chunks made 6 GETs and took 99.80 ms at the median, **$2.40 per
 million** at list prices: measured on S3 in-region, in the run of 2026-10-06, whose client had 128 sockets. On a spread layout it reads more
 bytes than it needs, which cross-region can cost more than the requests it saves. A segment's first `store.load()` made
-2 PUT + 3 GET, **$11.20 per million** at the default prices, pointer included: the requests are measured on S3 in-region, in the same run. A steady `store.load()` at `keep: 12` that deletes a generation by name made 2 PUT + 4 GET and a free delete, measured: **$11.60 per million**. A cold `count()` is one
+2 PUT + 3 GET, **$11.20 per million** at the default prices, pointer included: the requests are measured on S3 in-region, in the same run. A steady `store.load()` at `keep: 12` that deletes a generation by name costs **$12.00 per million**, for 2 PUT + 5 GET and a free delete, counted by the engine's tests. A cold `count()` is one
 pointer read: it reads no payload and no object.
 
 The trade is stated plainly. A membership check that misses the cache costs a ranged GET against object storage,

package/dist/bin/export-segments.d.ts

@@ -6,7 +6,7 @@ export interface ExportConfig {
     readonly format: ExportFormat;
     readonly namespace?: string;
 }
-/** Parse + validate config from an environment map. Throws a clear `Error` on misconfiguration. */
+/** Parse + validate config from an environment map. Throws a {@link ValidationError} on misconfiguration. */
 export declare function parseConfig(env: Record<string, string | undefined>): ExportConfig;
 /**
  * A filesystem {@link ExportSink}. Writes each segment to a **unique** `<segment><ext>.<uuid>.part` temp
@@ -15,6 +15,10 @@ export declare function parseConfig(env: Record<string, string | undefined>): Ex
  * another. Uses a `FileHandle` (not `createWriteStream`) so an I/O fault **rejects the write** rather than
  * emitting an unhandled `'error'` event that would crash the process, and loops on short writes so a partial
  * `write()` never truncates the output. Artifacts are **owner-only** (dir `0o700`, files `0o600`) — cleartext.
+ *
+ * A namespace named `manifest.json`, in any letter case, is refused with a `ValidationError`, so each of its segments
+ * lands in `failed`: its directory would take the manifest's path. Aborting a segment also removes its namespace
+ * directory when that leaves it empty.
  */
 export declare function fsSink(out: string): ExportSink;
 /** CLI entry: build a LocalFs store, export every enumerated segment, then write the manifest. */