@cloudbitmaps/roaring 0.9.0 → 0.11.0

package/README.md

@@ -1,119 +1,109 @@
 # @cloudbitmaps/roaring
 
-**Distributed, cloud-native Roaring Bitmaps.** Query and intersect billion-scale integer sets over tiered cloud
-storage (RAM → NoSQL → object store) at a fraction of an always-on cache — with the fast, familiar in-memory API:
-`add`, `has`, `remove`, `count`, `intersect`.
+**Large sets of integer ids, stored in your own object storage and read from anywhere.** Load a set as one immutable
+file, then ask `has`, `count`, `iterate`, `intersect`, `union` and `andNot`, with no server or cache to run. Part of
+[CloudBitmaps](https://github.com/cloudbitmaps/cloudbitmaps); this is the package you import from.
 
-This is the **flagship flavor** of the [CloudBitmaps](https://github.com/cloudbitmaps/cloudbitmaps) family: the
-roaring codec (CRoaring, via `roaring`) plus the `CloudRoaring` facade, on top of the codec-agnostic
-[`@cloudbitmaps/core`](https://www.npmjs.com/package/@cloudbitmaps/core) engine — which arrives **transitively**,
-so this is the only package you install.
+> **ESM-only, Node ≥ 22.12.** Use `import`; for `require()`, Jest and TypeScript, see
+> [CommonJS, Jest and TypeScript](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/getting-started.md#commonjs-jest-and-typescript).
+
+> Pre-1.0: the API and the on-disk format can still change. These docs describe `main`, ahead of the npm release.
+
+## Install
 
 ```bash
-npm i @cloudbitmaps/roaring @aws-sdk/client-s3 @aws-sdk/client-dynamodb   # + only the SDK(s) you use
+pnpm add @cloudbitmaps/roaring              # the store, with in-memory and local-disk backends
+pnpm add @cloudbitmaps/s3                   # the storage you have: or @cloudbitmaps/gcs, or @cloudbitmaps/azure-blob
 ```
 
-```ts
-import { CloudRoaring } from '@cloudbitmaps/roaring';
-import { S3ColdDriver } from '@cloudbitmaps/roaring/s3';
-import { DynamoDbWarmDriver } from '@cloudbitmaps/roaring/dynamodb';
-```
+On pnpm 10 and later, allow the one build script first, or the package throws at `import` while the install exits 0. Put this in your `package.json`: `{ "pnpm": { "onlyBuiltDependencies": ["roaring"] } }`. npm and pnpm 9 need nothing extra. [Details](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/getting-started.md#cannot-find-module-buildreleaseroaringnode-after-a-successful-install).
 
-Every storage driver is re-exported on a matching subpath (`/s3`, `/gcs`, `/azure`, `/dynamodb`, `/postgres`,
-`/redis`, `/mongodb`, `/cassandra`, `/mysql`); each backend SDK is an optional peer dependency, so the main entry
-stays lean. Ships two CLIs: `compact-segments` and `export-segments`.
+## Use
 
-## Measured, not modeled
+No setup needed: `MemoryStorage` keeps everything in the process, which is right for a first look and for tests.
+Save this as `first-run.mjs` and run it with `node`:
 
-Benchmarked against **real** S3 + DynamoDB in `us-east-1`, not an emulator: **$0.75 per million** incremental
-writes, **$0.14 per million** `count()`s, **$5.88 per million** segment publishes — against an always-on Redis-HA
-line of **$346/month, standing**. DynamoDB capacity is taken from AWS's own `ConsumedCapacity` rather than
-estimated. `count()` on a published segment does **0 payload reads**, and intersecting two 2,000,000-id segments
-fetches only the shared chunks.
+```js
+import { CloudRoaring, MemoryStorage } from '@cloudbitmaps/roaring';
 
-The trade is stated plainly rather than buried: a membership check that misses the hot cache costs a network
-round trip to your warm tier, where an in-process RAM store costs a memory read. If you need a sub-millisecond
-p99 on a working set that fits a bounded hot cache, use Redis. If your sets are large, mostly read, and shouldn't cost $346/month to keep warm, use this.
+const store = new CloudRoaring({ storage: new MemoryStorage() });
 
-## Coming from Redis bitmaps?
+await store.load({ segment: 'shoppers' }, [5, 99_999, 1_234_567_890]); // a load replaces the segment
+await store.load({ segment: 'active' }, [5, 7, 1_234_567_890]);
+
+const shoppers = store.segment('shoppers');
+console.log(await shoppers.has(99_999)); // true
+console.log(await shoppers.count()); // 3
+for await (const id of shoppers.intersect([store.segment('active')])) console.log(id); // 5, then 1234567890
+```
+
+To persist, pass a different **backend** as `storage`. Nothing else changes:
+
+| Backend | From | Use it for |
+|---|---|---|
+| `MemoryStorage` | `@cloudbitmaps/roaring` | tests and a first look |
+| `LocalFsStorage('./.cloudbitmaps')` | `@cloudbitmaps/roaring` | one process on one folder: a laptop or a CI job |
+| `S3Storage({ bucket, prefix })` | `@cloudbitmaps/s3` | production: the validated one |
+| `GcsStorage({ bucket, prefix })` | `@cloudbitmaps/gcs` | Google Cloud Storage; outside the [validated envelope](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/ROADMAP.md#the-validated-envelope--whats-proven-and-what-isnt) |
+| `AzureBlobStorage({ connectionString, container })` | `@cloudbitmaps/azure-blob` | Azure Blob Storage; outside the [validated envelope](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/ROADMAP.md#the-validated-envelope--whats-proven-and-what-isnt) |
 
-**A durable alternative to Redis bitmaps for set-shaped work** — audiences, dedup, suppression, membership —
-where the sets are large, mostly idle, must survive a restart, and are written in **batches**. You are not giving
-up the bitmap either: past **4,096 ids** in a 65,536-id chunk — 6.25% of it — Roaring stores that chunk *as* a flat
-bit array, byte for byte what you have now. It just stops paying for the chunks you never wrote to.
+There is no `add` or `remove`: a segment changes only by loading a new generation. A load is a batch job, not a
+request handler. `store.exists(ref)` says whether a segment has been loaded.
 
-| Redis | Here |
+## Options
+
+`new CloudRoaring({ storage, ...options })`. Only `storage` is required. A key the store does not take is refused by
+name.
+
+| Option | What it holds |
 |---|---|
-| `SETBIT` / `GETBIT` | `add` / `addMany` · `remove` / `removeMany` · `has` |
-| `prior = SETBIT` | `claimMany(ids)` — adds a batch and returns only the ids that were **not** already there, so an *"already sent to this id?"* check is exactly-once |
-| `BITCOUNT` | `count()` — exact, served from the index with no payload reads |
-| `BITOP AND` / `OR` / `DIFF` | `intersect` / `union` / `andNot` — and `intersect` skips chunks that cannot contribute |
-| `EXPIRE` | **`setRetention(ref, { expiresAt })` + `retireExpired()`** — a per-segment expiry the writer sets, and a sweep **you** schedule (this library starts no timer, so it works the same in a Lambda and a server). No per-**id** TTL: a bitmap stores ids, not timestamps. And never enable your backend's own row expiry on the warm table — those rows are un-compacted deltas, so it discards writes silently |
-
-**It is not a drop-in replacement, and two limits are worth knowing before you port anything.**
-
-**1. Do not port a per-id write loop.** `SETBIT` flips one bit in place and is genuinely O(1). Here a warm write
-re-serializes a whole 65,536-id chunk, so **5,000 ids added one at a time cost 5,000 writes and 23,762 KB, against
-1 write and 8 KB for the same ids in one call** — roughly 3,000× the bytes. Batch and you are far cheaper than
-Redis; port the loop literally and you are far more expensive. On a per-request-metered store (DynamoDB on-demand)
-that is a line item; on an instance-priced one (ElastiCache, RDS) it surfaces as IOPS and latency headroom instead,
-which makes it easier to miss rather than cheaper. `claimMany` takes a batch for exactly this reason.
-
-**2. What does not carry over: the raw bytes.** A `.crbm` object is not a flat bit array, so anything reading your
-Redis bitmap's underlying string — a job that `GET`s the key and indexes into it, a byte-for-byte backup —
-will not read ours. `BITFIELD`, `BITPOS`, `BITOP NOT` and byte-range `BITCOUNT` have no equivalent either: this
-is a set of ids, not an addressable bit buffer, and `NOT` in particular has no bounded universe to complement
-against. Raw bit-position import/export is unbuilt;
-[say so in an issue](https://github.com/cloudbitmaps/cloudbitmaps/issues) if you need it, because that is what
-decides whether it gets built.
-
-## Retiring data: a per-segment expiry, and a sweep you schedule
-
-A **segment** can expire; an individual **id** cannot (a bitmap stores ids, not `(id, timestamp)` pairs — a
-timestamp per id costs more than the compression saves).
-
-```ts
-const DAY = 86_400_000;
-const ref = { namespace: 'active-daily', segment: '2026-08-05' };
-
-await store.setRetention(ref, { expiresAt: Date.now() + 30 * DAY }); // once, when you create the bucket
-```
+| `storage` (required) | a backend (see above) |
+| `cache` | `maxChunks`, `genTtlMs` (how soon a reader sees a new load, default 2000 ms), `readerMax`, `readerMaxBytes` |
+| `encryption` | `{ keystore, required }`: encrypt segments at rest with your own key |
+| `retry` | tune the retry of reads, or `false` to turn it off |
+| `metrics` | a metrics sink; off by default |
+| `budget` | `{ maxRequests }` or `false`: the per-call request ceiling |
+| `seams` | `{ clock, rng }`: determinism, for tests |
 
-`expiresAt` is an absolute instant **you** compute, not a duration derived from anything observed: compaction
-rewrites every basis such a duration could use, so "30 days since the last write" would keep a busy bucket alive
-precisely *because* the daemon was keeping it cheap.
+## What it costs
 
-Then, from whatever schedule your deployment already has — an EventBridge rule, a Kubernetes `CronJob`, `cron`, a
-queue job — run the sweep. **This library starts no background timer**, deliberately: the same code has to behave
-identically in a Lambda, an edge isolate and a long-lived server, and a timer that only works in one of those is
-worse than none.
+Measured against real S3 in `us-east-1`, with the pointer in the same bucket as the data: the median cold `intersect`
+of two 500,000-id segments sharing 100 of their 1,999 chunks made 206 GETs, **$82.40 per million**, requesting only
+the shared chunks. Inside the region it is expected at 204 GETs, $81.60 per million. Writing and publishing a segment
+is **$11.20 per million**, pointer included, from its measured requests at list prices. `count()` reads no payload.
 
-```ts
-const swept = await store.retireExpired({ namespace: 'active-daily' });
-for (const e of swept.entries) if (e.action === 'skipped') console.warn(e.segment, e.reason);
-if (swept.limited) scheduleAnotherPassSoon(); // the per-cycle cap deferred some; re-run
-```
+The trade is stated plainly. A membership check that misses the cache costs a ranged GET against object storage,
+where an in-process store costs a memory read. If you need sub-millisecond answers on a working set that fits a
+bounded cache, use Redis. [The benchmarks](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/benchmarks.md)
+give the method and what the numbers do not establish.
 
-Each retirement goes through `dropSegment`, so the Warm → registry → Cold ordering is one implementation rather
-than two. The sweep is bounded (`limit`, default 100), previewable (`dryRun`), and returns a per-segment ledger
-rather than throwing — a fault on one segment must not decide the fate of the other ninety-nine. Once a day is
-enough for daily buckets. Full walkthrough:
-[getting-started §13.5](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/getting-started.md#135-retention-ttl-and-pruning--what-exists-and-what-doesnt).
+## Before production
 
-## No seed step, and compaction is optional
+- Grant the permissions the library issues and add a lifecycle rule to abort incomplete uploads.
+- Use a bucket that honors conditional writes, and `@aws-sdk/client-s3` 3.645.0 or later if you pass your own S3 client.
+- Set a request timeout on your storage client. Reads retry by themselves; a write that fails transiently is yours to re-run.
+- Back up the registry with the data, and the keystore too if you encrypt. If you lose the key, the data is gone.
+- Schedule your loads, and `retireExpired` if you use retention. Run loads off the request path.
+
+The [production checklist](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/production.md) covers each item.
+
+## Coming from Redis bitmaps?
 
-A brand-new segment is usable the moment you construct the store: `addMany`, `has`, `remove`, `count`, `iterate`
-and the whole set algebra work against a segment with **no cold generation and no registry row**, because a read
-merges `(cold ∪ warm.adds) \ warm.removes` and an absent cold tier just makes that merge trivial. Bulk-load is an
-**import** path for data you already have elsewhere, not an initialization step.
+`GETBIT` is `has`, `BITCOUNT` is `count`, and `BITOP AND` / `OR` / `DIFF` are `intersect` / `union` / `andNot`, with
+`intersectInto`, `unionInto` and `andNotInto` publishing the result as a new generation. `EXPIRE` becomes
+`setRetention` plus a `retireExpired` sweep you schedule, per segment, never per id. `SETBIT` has no equivalent:
+there is no per-id write. Build the set upstream and load it, and do not loop one id at a time. Remove one id
+everywhere with `eraseSubject`, a rewrite for compliance, not a hot-path verb. This is not a drop-in replacement, and
+what does not carry over is the write model and the raw bytes: `exportSegments` writes portable Roaring and ndjson, not Redis's layout. [The full mapping](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/getting-started.md#coming-from-redis-bitmaps)
+says what else differs.
 
-Compaction is a **cost** optimization, never a correctness requirement — it buys cheaper storage, smaller rewrites,
-and a free index-only `count()`. For a write-once dated bucket you retire (with `dropSegment`, or by recording an
-expiry as above), skip it entirely.
+## Documentation
 
-Redis stays first-class as a **warm tier** underneath this (`@cloudbitmaps/roaring/redis`) — the point above is
-about replacing `SETBIT`-on-one-giant-key as your *data model*, not replacing Redis as infrastructure.
+- [Getting started](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/getting-started.md)
+- [Before production](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/production.md)
+- [API reference](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/api-reference.md)
+- [Changelog](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/CHANGELOG.md)
+- [Privacy and shared responsibility](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/PRIVACY.md)
+## License
 
-Full README, guides, [benchmarks](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/benchmarks.md) (with
-the method and what the numbers do *not* establish), and the design corpus live in the
-[repository](https://github.com/cloudbitmaps/cloudbitmaps). Licensed Apache-2.0.
+Apache-2.0

package/dist/bin/export-segments.d.ts

@@ -0,0 +1,21 @@
+#!/usr/bin/env node
+import type { ExportFormat, ExportManifest, ExportSink } from '../index.js';
+export interface ExportConfig {
+    readonly root: string;
+    readonly out: string;
+    readonly format: ExportFormat;
+    readonly namespace?: string;
+}
+/** Parse + validate config from an environment map. Throws a clear `Error` on misconfiguration. */
+export declare function parseConfig(env: Record<string, string | undefined>): ExportConfig;
+/**
+ * A filesystem {@link ExportSink}. Writes each segment to a **unique** `<segment><ext>.<uuid>.part` temp
+ * (`O_EXCL` create, mode `0o600`) and atomically renames it into `<out>/<namespace>/<segment><ext>` on `close()`,
+ * so a half-written file never masquerades as complete and concurrent exports to the same dir can't clobber one
+ * another. Uses a `FileHandle` (not `createWriteStream`) so an I/O fault **rejects the write** rather than
+ * emitting an unhandled `'error'` event that would crash the process, and loops on short writes so a partial
+ * `write()` never truncates the output. Artifacts are **owner-only** (dir `0o700`, files `0o600`) — cleartext.
+ */
+export declare function fsSink(out: string): ExportSink;
+/** CLI entry: build a LocalFs store, export every enumerated segment, then write the manifest. */
+export declare function main(env?: Record<string, string | undefined>, now?: () => number): Promise<ExportManifest>;