@cloudbitmaps/roaring 0.8.2 → 0.10.0

package/README.md

@@ -1,84 +1,182 @@
 # @cloudbitmaps/roaring
 
-**Distributed, cloud-native Roaring Bitmaps.** Query and intersect billion-scale integer sets over tiered cloud
-storage (RAM → NoSQL → object store) at a fraction of an always-on cache — with the fast, familiar in-memory API:
-`add`, `has`, `remove`, `count`, `intersect`.
+> **ESM-only, Node ≥ 22.12.** This package ships as ES modules; there is no CommonJS bundle.
+> `require()` works on Node 22.12+ through `require(esm)`, but a runner with its own CommonJS loader
+> (notably Jest in its default configuration) does not get that and needs `import` instead. On TypeScript,
+> a CommonJS project needs `"module": "nodenext"` or `"node20"`. See the
+> [repository README](https://github.com/cloudbitmaps/cloudbitmaps#install--entry-points) for the details.
+
+
+**Distributed, cloud-native Roaring Bitmaps.** Query and intersect billion-scale integer sets straight out of
+object storage — a bounded RAM cache over immutable `.crbm` objects in your own bucket — at a fraction of an
+always-on cache, with the familiar bitmap API: `has`, `count`, `iterate`, `intersect`, `union`, `andNot`.
 
 This is the **flagship flavor** of the [CloudBitmaps](https://github.com/cloudbitmaps/cloudbitmaps) family: the
 roaring codec (CRoaring, via `roaring`) plus the `CloudRoaring` facade, on top of the codec-agnostic
-[`@cloudbitmaps/core`](https://www.npmjs.com/package/@cloudbitmaps/core) engine — which arrives **transitively**,
-so this is the only package you install.
+[`@cloudbitmaps/core`](https://www.npmjs.com/package/@cloudbitmaps/core) engine, which is a dependency of
+this package and of the storage package you pair it with — so you never install or name core yourself.
 
 ```bash
-npm i @cloudbitmaps/roaring @aws-sdk/client-s3 @aws-sdk/client-dynamodb   # + only the SDK(s) you use
+pnpm add @cloudbitmaps/roaring @cloudbitmaps/s3   # the codec, and the storage you have
+# npm i @cloudbitmaps/roaring @cloudbitmaps/s3     # the same, with npm
 ```
 
+> **On pnpm 10+, allow the one build script.** pnpm 10 skips dependency build scripts by default, so
+> the `roaring` native addon never downloads and the package throws at `import` — while the install
+> itself prints a warning and **exits 0**. Add this to your `package.json`, then install:
+>
+> ```json
+> { "pnpm": { "onlyBuiltDependencies": ["roaring"] } }
+> ```
+>
+> pnpm 9 and npm run it already. [Full symptoms and fixes](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/getting-started.md#cannot-find-module-buildreleaseroaringnode-after-a-successful-install).
+
 ```ts
 import { CloudRoaring } from '@cloudbitmaps/roaring';
-import { S3ColdDriver } from '@cloudbitmaps/roaring/s3';
-import { DynamoDbWarmDriver } from '@cloudbitmaps/roaring/dynamodb';
+import { S3Storage } from '@cloudbitmaps/s3';
 ```
 
-Every storage driver is re-exported on a matching subpath (`/s3`, `/gcs`, `/azure`, `/dynamodb`, `/postgres`,
-`/redis`, `/mongodb`, `/cassandra`, `/mysql`); each backend SDK is an optional peer dependency, so the main entry
-stays lean. Ships two CLIs: `compact-segments` and `export-segments`.
+Install the storage you have alongside this: **`@cloudbitmaps/s3`** (also R2, MinIO, Ceph, Wasabi, B2),
+**`@cloudbitmaps/gcs`** or **`@cloudbitmaps/azure-blob`**. Each depends on its cloud SDK for real, so nothing
+is an optional peer and no install carries an SDK you do not use. Ships one CLI: `export-segments`.
+
+## The model in one paragraph
+
+A segment is a set of **write-once `.crbm` generations** in your bucket behind one small registry row saying which
+generation is current. Data gets in by **loading** a new generation — you compute the set upstream (a warehouse
+query, a batch job, a combine of other segments) and `store.load` streams it into a single immutable object,
+checks the result is plausible, then advances the pointer forward-only. Nothing mutates a stored bitmap, so a crash before the publish
+leaves the previous generation authoritative, a rerun is idempotent, and there is no partially-visible write.
+Reads (`has`, `count`, `iterate`, `intersect`, `union`, `andNot`) see one whole, checksum-verified generation.
+
+```ts
+// One bucket is the whole deployment: the generations and the pointer, stated once.
+const backend = new S3Storage({ bucket: 'bitmaps', region: 'us-east-1' });
+
+const store = new CloudRoaring({ storage: backend });
+
+// One call is the whole write path: write the object, check it, move the pointer, collect the old generation.
+await store.load({ segment: 'high-value' }, idsFromWarehouse());
+
+const seg = store.segment('high-value');
+
+await seg.has(1_234_567_890); // one chunk — from the cache after the first read
+await seg.count(); // exact, and summed from the index with 0 payload reads
+for await (const id of seg.intersect([store.segment('eu-residents')], { exclude: [store.segment('opted-out')] })) {
+  /* streamed ascending; only the chunks that can contribute are ever fetched */
+}
+```
+
+`intersectInto`, `unionInto` and `andNotInto` write the result as a **new generation of the destination** rather
+than streaming it to you, using the same write-once-then-publish protocol.
 
 ## Measured, not modeled
 
-Benchmarked against **real** S3 + DynamoDB in `us-east-1`, not an emulator: **$0.75 per million** incremental
-writes, **$0.14 per million** `count()`s, **$5.88 per million** segment publishes — against an always-on Redis-HA
-line of **$346/month, standing**. DynamoDB capacity is taken from AWS's own `ConsumedCapacity` rather than
-estimated. `count()` on a published segment does **0 payload reads**, and intersecting two 2,000,000-id segments
-fetches only the shared chunks.
+Benchmarked against **real** S3 in `us-east-1`, not an emulator: **$0.14 per million** `count()`s and
+**$5.88 per million** segment publishes — against an always-on Redis-HA line of **$346/month, standing**, and
+**$0.03/month** for 1.2 GiB of segments at rest. Request counts are read off the AWS SDK layer rather than
+estimated from sizes. `count()` on a published segment does **0 payload reads**, and intersecting two
+2,000,000-id segments fetches only the shared chunks. That run kept the generation pointer in a NoSQL
+registry that no longer ships, so both figures are the object-store half rather than today's total — the
+pointer is now an object request that is not in them.
 
-The trade is stated plainly rather than buried: a membership check that misses the hot cache costs a network
-round trip to your warm tier, where an in-process RAM store costs a memory read. If you need a sub-millisecond
-p99 on a working set that fits a bounded hot cache, use Redis. If your sets are large, mostly read, and shouldn't cost $346/month to keep warm, use this.
+The trade is stated plainly rather than buried: a membership check that misses the cache costs a ranged GET
+against object storage, where an in-process RAM store costs a memory read. If you need a sub-millisecond p99 on a
+working set that fits a bounded cache, use Redis. If your sets are large, mostly read, and shouldn't cost
+$346/month to keep warm, use this.
 
 ## Coming from Redis bitmaps?
 
-**A durable alternative to Redis bitmaps for set-shaped work** — audiences, dedup, suppression, membership —
-where the sets are large, mostly idle, must survive a restart, and are written in **batches**. You are not giving
-up the bitmap either: past **4,096 ids** in a 65,536-id chunk — 6.25% of it — Roaring stores that chunk *as* a flat
+**A durable home for set-shaped work** — audiences, dedup, suppression, membership, eligibility — where the sets
+are large, mostly read, must survive a restart, and are **computed in batches** upstream. You are not giving up
+the bitmap either: past **4,096 ids** in a 65,536-id chunk — 6.25% of it — Roaring stores that chunk *as* a flat
 bit array, byte for byte what you have now. It just stops paying for the chunks you never wrote to.
 
+The read side carries over one-for-one:
+
 | Redis | Here |
 |---|---|
-| `SETBIT` / `GETBIT` | `add` / `addMany` · `remove` / `removeMany` · `has` |
-| `prior = SETBIT` | `claimMany(ids)` — adds a batch and returns only the ids that were **not** already there, so an *"already sent to this id?"* check is exactly-once |
+| `GETBIT` | `has(id)` |
 | `BITCOUNT` | `count()` — exact, served from the index with no payload reads |
-| `BITOP AND` / `OR` / `DIFF` | `intersect` / `union` / `andNot` — and `intersect` skips chunks that cannot contribute |
-| `EXPIRE` | a dated segment + `dropSegment` — there is no TTL, and enabling your backend's own row expiry on the warm table would discard un-compacted deltas |
-
-**It is not a drop-in replacement, and two limits are worth knowing before you port anything.**
-
-**1. Do not port a per-id write loop.** `SETBIT` flips one bit in place and is genuinely O(1). Here a warm write
-re-serializes a whole 65,536-id chunk, so **5,000 ids added one at a time cost 5,000 writes and 23,762 KB, against
-1 write and 8 KB for the same ids in one call** — roughly 3,000× the bytes. Batch and you are far cheaper than
-Redis; port the loop literally and you are far more expensive. On a per-request-metered store (DynamoDB on-demand)
-that is a line item; on an instance-priced one (ElastiCache, RDS) it surfaces as IOPS and latency headroom instead,
-which makes it easier to miss rather than cheaper. `claimMany` takes a batch for exactly this reason.
-
-**2. What does not carry over: the raw bytes.** A `.crbm` object is not a flat bit array, so anything reading your
-Redis bitmap's underlying string — a job that `GET`s the key and indexes into it, a byte-for-byte backup —
-will not read ours. `BITFIELD`, `BITPOS`, `BITOP NOT` and byte-range `BITCOUNT` have no equivalent either: this
-is a set of ids, not an addressable bit buffer, and `NOT` in particular has no bounded universe to complement
-against. Raw bit-position import/export is unbuilt;
+| `BITOP AND` / `OR` / `DIFF` | `intersect` / `union` / `andNot` — streamed, and `intersect` skips chunks that cannot contribute. `intersectInto` / `unionInto` / `andNotInto` publish the result as a new generation of the destination |
+| `SETBIT` | **no equivalent — there is no per-id write.** Build the set upstream and load it; remove one id everywhere with `eraseSubject` (a generation rewrite for compliance, not a hot-path verb) |
+| `EXPIRE` | **`setRetention(ref, { expiresAt })` + `retireExpired()`** — a per-segment expiry the writer sets, and a sweep **you** schedule (this library starts no timer, so it works the same in a Lambda and a server). No per-**id** TTL: a bitmap stores ids, not timestamps |
+
+**It is not a drop-in replacement, and the difference is the write model.** Redis mutates one bit in place per
+call; here a segment changes only by getting a new generation. So a `SETBIT` loop has nothing to port to: if your
+set is defined by a query, run the query and load the result; if it is defined by events arriving one at a time,
+accumulate them where they arrive (Redis is good at that) and load the set on a cadence. A load bills per
+**object**, not per id — ten million ids are a handful of PUTs.
+
+**What does not carry over: the raw bytes.** A `.crbm` object is not a flat bit array, so anything reading your
+Redis bitmap's underlying string — a job that `GET`s the key and indexes into it, a byte-for-byte backup — will
+not read ours. `BITFIELD`, `BITPOS`, `BITOP NOT` and byte-range `BITCOUNT` have no equivalent either: this is a
+set of ids, not an addressable bit buffer, and `NOT` in particular has no bounded universe to complement against.
+Raw bit-position import/export is unbuilt;
 [say so in an issue](https://github.com/cloudbitmaps/cloudbitmaps/issues) if you need it, because that is what
 decides whether it gets built.
 
-## No seed step, and compaction is optional
+## Compliance is built in, not bolted on
+
+- `subjectReport(id)` — which segments an id is in (GDPR Art. 15).
+- `eraseSubject(id)` — rewrites every segment holding the id without it and deletes the generation that held the
+  bit, so it is **physically gone from the bucket when the call returns**; you get an erasure ledger back and a
+  `segment.rewrite` audit event per segment it had to rewrite (Art. 17; an id erased out of a retained
+  superseded generation is collected rather than rewritten, so that one is attested by the ledger entry alone).
+- `destroySegment` / `eraseNamespace` — **crypto-shred**: drop the segment's wrapped data key so its encrypted
+  bytes are unreadable *everywhere, including backups*.
+- `dropSegment(ref, { confirmSegment, dryRun })` — retire a segment and reclaim its storage.
+- `exists(ref)` — whether this segment's pointer resolves a generation, as one registry read. There is no
+  `create`: `segment(name)` is a validated address and does no I/O, so a name can never collide. Not the
+  same as `count() > 0` — a segment loaded with no ids exists and counts zero.
+- `segments({ namespace })` — everything the registry holds, streamed, so you do not keep your own list of
+  segment names beside the store. An admin/dashboard call, not a request-path one.
+- `checkConsistency()` — after a restore, verify every pointer's object is actually present.
+- `exportSegments(sink, { format })` — eject every segment to portable `roaring` or `ndjson`. Your exit path.
+
+## Retiring data: a per-segment expiry, and a sweep you schedule
+
+A **segment** can expire; an individual **id** cannot (a bitmap stores ids, not `(id, timestamp)` pairs — a
+timestamp per id costs more than the compression saves).
+
+```ts
+const DAY = 86_400_000;
+const ref = { namespace: 'active-daily', segment: '2026-08-05' };
+
+await store.setRetention(ref, { expiresAt: Date.now() + 30 * DAY }); // once, when you create the bucket
+```
+
+`expiresAt` is an absolute instant **you** compute, not a duration derived from anything observed: every load
+rewrites the basis such a duration would use, so "30 days since the last write" would keep a busy bucket alive
+precisely *because* it is being refreshed. `getRetention` reads the policy back and `clearRetention` cancels it.
 
-A brand-new segment is usable the moment you construct the store: `addMany`, `has`, `remove`, `count`, `iterate`
-and the whole set algebra work against a segment with **no cold generation and no registry row**, because a read
-merges `(cold ∪ warm.adds) \ warm.removes` and an absent cold tier just makes that merge trivial. Bulk-load is an
-**import** path for data you already have elsewhere, not an initialization step.
+Then, from whatever schedule your deployment already has — an EventBridge rule, a Kubernetes `CronJob`, `cron`, a
+queue job — run the sweep. **This library starts no background timer**, deliberately: the same code has to behave
+identically in a Lambda, an edge isolate and a long-lived server, and a timer that only works in one of those is
+worse than none.
 
-Compaction is a **cost** optimization, never a correctness requirement — it buys cheaper storage, smaller rewrites,
-and a free index-only `count()`. For a write-once dated bucket you retire with `dropSegment`, skip it entirely.
+```ts
+const swept = await store.retireExpired({ namespace: 'active-daily' });
+for (const e of swept.entries) if (e.action === 'skipped') console.warn(e.segment, e.reason);
+if (swept.limited) scheduleAnotherPassSoon(); // the per-cycle cap deferred some; re-run
+```
 
-Redis stays first-class as a **warm tier** underneath this (`@cloudbitmaps/roaring/redis`) — the point above is
-about replacing `SETBIT`-on-one-giant-key as your *data model*, not replacing Redis as infrastructure.
+Each retirement goes through `dropSegment`, so the registry → object-store ordering is one implementation rather
+than two. The sweep is bounded (`limit`, default 100), previewable (`dryRun`), and returns a per-segment ledger
+rather than throwing — a fault on one segment must not decide the fate of the other ninety-nine. Once a day is
+enough for daily buckets. Full walkthrough:
+[getting-started §13.5](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/getting-started.md#135-retention-ttl-and-pruning--what-exists-and-what-doesnt).
+
+## No background process, and nothing to seed
+
+There is no daemon, no compaction pass and no lifecycle worker to run: a segment exists once you have loaded a
+generation into it, and the only scheduled work is the retention sweep above (plus `gcOrphanGenerations` if you
+want superseded generations collected sooner than the sweep does it). Pass a **backend** — `S3Storage`,
+`GcsStorage` or `AzureBlobStorage` from the storage package you installed, or `LocalFsStorage` /
+`MemoryStorage` from this one — and you get all of it: generations resolved with one strong read, encrypted
+segments, and the lifecycle helpers. `storage` also accepts a bare driver or a pre-built chunk source for
+read-only wiring, which carries no registry and so offers none of those. There is no separate `registry`
+option any more; passing one is refused by name.
 
 Full README, guides, [benchmarks](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/benchmarks.md) (with
 the method and what the numbers do *not* establish), and the design corpus live in the

package/dist/bin/export-segments.d.ts

@@ -0,0 +1,21 @@
+#!/usr/bin/env node
+import type { ExportFormat, ExportManifest, ExportSink } from '../index.js';
+export interface ExportConfig {
+    readonly root: string;
+    readonly out: string;
+    readonly format: ExportFormat;
+    readonly namespace?: string;
+}
+/** Parse + validate config from an environment map. Throws a clear `Error` on misconfiguration. */
+export declare function parseConfig(env: Record<string, string | undefined>): ExportConfig;
+/**
+ * A filesystem {@link ExportSink}. Writes each segment to a **unique** `<segment><ext>.<uuid>.part` temp
+ * (`O_EXCL` create, mode `0o600`) and atomically renames it into `<out>/<namespace>/<segment><ext>` on `close()`,
+ * so a half-written file never masquerades as complete and concurrent exports to the same dir can't clobber one
+ * another. Uses a `FileHandle` (not `createWriteStream`) so an I/O fault **rejects the write** rather than
+ * emitting an unhandled `'error'` event that would crash the process, and loops on short writes so a partial
+ * `write()` never truncates the output. Artifacts are **owner-only** (dir `0o700`, files `0o600`) — cleartext.
+ */
+export declare function fsSink(out: string): ExportSink;
+/** CLI entry: build a LocalFs store, export every enumerated segment, then write the manifest. */
+export declare function main(env?: Record<string, string | undefined>, now?: () => number): Promise<ExportManifest>;