@cloudbitmaps/core 0.8.2 → 0.10.0

package/NOTICE

@@ -1,4 +1,4 @@
-CloudRoaring
+@cloudbitmaps/core
 Copyright 2026 Sharvil Kadam
 
 This product includes software developed as an open-source project.
@@ -7,6 +7,6 @@ file. You may obtain a copy of the License at:
 
     http://www.apache.org/licenses/LICENSE-2.0
 
-CloudRoaring builds on the Roaring Bitmap data structure and is designed to use
-the CRoaring engine (via the `roaring` Node.js binding) for in-memory bit math.
-Those projects are independent works under their own licenses.
+@cloudbitmaps/core is the codec-agnostic engine of CloudBitmaps. It contains no bitmap
+codec and no cloud SDK, and has zero runtime dependencies; a flavor package supplies the
+codec through the CodecInterface seam.

package/README.md

@@ -1,20 +1,42 @@
 # @cloudbitmaps/core
 
+> **ESM-only, Node ≥ 22.12.** This package ships as ES modules; there is no CommonJS bundle.
+> `require()` works on Node 22.12+ through `require(esm)`, but a runner with its own CommonJS loader
+> (notably Jest in its default configuration) does not get that and needs `import` instead. On TypeScript,
+> a CommonJS project needs `"module": "nodenext"` or `"node20"`. See the
+> [repository README](https://github.com/cloudbitmaps/cloudbitmaps#install--entry-points) for the details.
+
+
 The **codec-agnostic cloud engine** behind the [CloudBitmaps](https://github.com/cloudbitmaps/cloudbitmaps) family:
-tiered storage (HOT RAM → WARM key/value deltas → COLD immutable `.crbm` objects), serverless chunk-skipping
-intersection, the segment registry, crash-safe 2-phase-commit compaction, encryption-at-rest + crypto-shred, and
-**every storage driver** (S3 · GCS · Azure Blob · DynamoDB · PostgreSQL · Redis · MongoDB · Cassandra/ScyllaDB ·
-MySQL/MariaDB, each on its own subpath with the backend SDK as an optional peer dependency).
+a bounded RAM cache over immutable `.crbm` objects in STORAGE, serverless chunk-skipping intersection, the segment
+registry, the write-once load-and-publish write path, generation GC, encryption-at-rest + crypto-shred, subject
+erasure by generation rewrite, segment lifecycle (disposal, and a per-segment retention policy with the sweep
+that enforces it), and the **in-memory and local-filesystem drivers**. It contains **no cloud SDK at all**: the
+cloud drivers are their own packages (`@cloudbitmaps/s3` · `/gcs` · `/azure-blob`), each hosting both the
+generations and the registry, and each built against `@cloudbitmaps/core/driver-kit` — the declared contract a
+driver package depends on, which a third-party driver can use too.
 
 ## You probably want a flavor, not this package
 
 This package holds no bitmap codec — that lives in a *flavor* package which depends on this one and supplies it
-through the `CodecInterface` seam. Install the flavor; `@cloudbitmaps/core` arrives **transitively**:
+through the `CodecInterface` seam, and it holds no cloud driver either — those are packages of their own.
+Install a codec and a storage; `@cloudbitmaps/core` is a dependency of both and never named by you:
 
 ```bash
-npm i @cloudbitmaps/roaring        # the roaring flavor (flagship)
+pnpm add @cloudbitmaps/roaring @cloudbitmaps/s3   # the roaring flavor (flagship), and the storage you have
+# npm i @cloudbitmaps/roaring @cloudbitmaps/s3     # the same, with npm
 ```
 
+> **On pnpm 10+, allow the one build script.** pnpm 10 skips dependency build scripts by default, so
+> the `roaring` native addon never downloads and the package throws at `import` — while the install
+> itself prints a warning and **exits 0**. Add this to your `package.json`, then install:
+>
+> ```json
+> { "pnpm": { "onlyBuiltDependencies": ["roaring"] } }
+> ```
+>
+> pnpm 9 and npm run it already. [Full symptoms and fixes](https://github.com/cloudbitmaps/cloudbitmaps/blob/main/docs/guide/getting-started.md#cannot-find-module-buildreleaseroaringnode-after-a-successful-install).
+
 Depend on `@cloudbitmaps/core` directly only to **author a flavor or a driver**. It has **zero runtime
 dependencies** of its own.
 

package/dist/chunk-BUPDGZRR.js

@@ -0,0 +1,461 @@
+// src/core/errors.ts
+var ERROR_BRAND = /* @__PURE__ */ Symbol.for("cloud-roaring.error");
+var TRANSIENT_BRAND = /* @__PURE__ */ Symbol.for("cloud-roaring.error.transient");
+var CloudRoaringError = class extends Error {
+  /** Cross-bundle brand — see the predicates ({@link isCloudRoaringError}, …). Non-enumerable-ish (symbol key ⇒ not in JSON). */
+  [ERROR_BRAND] = true;
+  constructor(message) {
+    super(message);
+    this.name = new.target.name;
+  }
+};
+var ValidationError = class extends CloudRoaringError {
+};
+var WriteConflictError = class extends CloudRoaringError {
+};
+var IntegrityError = class extends CloudRoaringError {
+};
+var NotFoundError = class extends CloudRoaringError {
+};
+var UnsupportedError = class extends CloudRoaringError {
+};
+var CapabilityError = class extends CloudRoaringError {
+};
+var BudgetExceededError = class extends CloudRoaringError {
+};
+var KeyUnavailableError = class extends CloudRoaringError {
+};
+var TransientError = class extends CloudRoaringError {
+  /** A second brand so the whole transient subtree (incl. {@link TimeoutError}) is classifiable cross-bundle. */
+  [TRANSIENT_BRAND] = true;
+  constructor(message, options) {
+    super(message);
+    if (options && "cause" in options) this.cause = options.cause;
+  }
+};
+var TimeoutError = class extends TransientError {
+};
+function hasBrand(err, brand) {
+  return typeof err === "object" && err !== null && err[brand] === true;
+}
+function isCloudRoaringError(err) {
+  return hasBrand(err, ERROR_BRAND);
+}
+function isWriteConflictError(err) {
+  return isCloudRoaringError(err) && err.name === "WriteConflictError";
+}
+function isTransientError(err) {
+  return hasBrand(err, TRANSIENT_BRAND);
+}
+function isNotFoundError(err) {
+  return isCloudRoaringError(err) && err.name === "NotFoundError";
+}
+function isIntegrityError(err) {
+  return isCloudRoaringError(err) && err.name === "IntegrityError";
+}
+function isValidationError(err) {
+  return isCloudRoaringError(err) && err.name === "ValidationError";
+}
+
+// src/core/name-codec.ts
+var KEY_SAFE = /[A-Za-z0-9._:-]/;
+var PATH_SAFE = /[A-Za-z0-9._-]/;
+var UTF8 = new TextEncoder();
+function escapeChar(ch) {
+  let out = "";
+  for (const byte of UTF8.encode(ch)) {
+    out += `%${byte.toString(16).toUpperCase().padStart(2, "0")}`;
+  }
+  return out;
+}
+function encodeWith(name, safe) {
+  let out = "";
+  for (const ch of name) {
+    out += ch !== "%" && safe.test(ch) ? ch : escapeChar(ch);
+  }
+  return out.startsWith("_") ? `%5F${out.slice(1)}` : out;
+}
+function decodePercent(encoded) {
+  const bytes = [];
+  for (let i = 0; i < encoded.length; ) {
+    if (encoded[i] === "%" && i + 2 < encoded.length + 1) {
+      const hex = encoded.slice(i + 1, i + 3);
+      if (!/^[0-9A-Fa-f]{2}$/.test(hex)) return encoded;
+      bytes.push(parseInt(hex, 16));
+      i += 3;
+    } else {
+      for (const b of new TextEncoder().encode(encoded[i])) bytes.push(b);
+      i += 1;
+    }
+  }
+  return new TextDecoder("utf-8", { fatal: false, ignoreBOM: true }).decode(new Uint8Array(bytes));
+}
+function encodeNameForKey(name) {
+  return defuseDotComponent(encodeWith(name, KEY_SAFE));
+}
+function defuseDotComponent(encoded) {
+  return encoded === "." || encoded === ".." ? encoded.replaceAll(".", "%2E") : encoded;
+}
+function decodeNameFromKey(encoded) {
+  return decodePercent(encoded);
+}
+function encodeNameForPath(name) {
+  let encoded = encodeWith(name, PATH_SAFE);
+  const dotted = defuseDotComponent(encoded);
+  if (dotted !== encoded) return dotted;
+  const stem = encoded.split(".")[0] ?? "";
+  if (/^(?:CON|PRN|AUX|NUL|COM[1-9]|LPT[1-9])$/i.test(stem)) {
+    encoded = escapeChar(encoded[0]) + encoded.slice(1);
+  }
+  if (encoded.endsWith(".")) encoded = `${encoded.slice(0, -1)}%2E`;
+  return encoded;
+}
+function decodeNameFromPath(encoded) {
+  return decodePercent(encoded);
+}
+
+// src/core/validate.ts
+var CHUNK_KEY_MAX = 65535;
+var MAX_NAME_LENGTH = 256;
+var LONE_SURROGATE = /[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?<![\uD800-\uDBFF])[\uDC00-\uDFFF]/;
+function encodedNameLength(value) {
+  return Math.max(encodeNameForKey(value).length, encodeNameForPath(value).length);
+}
+function validatePart(value, field) {
+  if (typeof value !== "string") {
+    throw new ValidationError(`${field} must be a string; got ${typeof value}`);
+  }
+  if (value.length === 0) {
+    throw new ValidationError(`${field} must not be empty`);
+  }
+  if (value.length > MAX_NAME_LENGTH) {
+    throw new ValidationError(
+      `${field} is too long: ${value.length} characters (limit ${MAX_NAME_LENGTH} once encoded for a storage key)`
+    );
+  }
+  if (LONE_SURROGATE.test(value)) {
+    throw new ValidationError(
+      `${field} contains an unpaired surrogate, so it has no UTF-8 encoding and cannot be stored. This is usually a string sliced through an astral character (an emoji, say) \u2014 slice by code point instead.`
+    );
+  }
+  const encoded = encodedNameLength(value);
+  if (encoded > MAX_NAME_LENGTH) {
+    throw new ValidationError(
+      `${field} is too long: ${encoded} characters once encoded for storage (limit ${MAX_NAME_LENGTH}). The name itself is ${value.length} characters \u2014 encoding expands anything outside [A-Za-z0-9._-], so a name of mostly non-ASCII text, or one full of colons, reaches the limit sooner than its length suggests.`
+    );
+  }
+}
+function validateSegmentRef(ref) {
+  validatePart(ref.segment, "segment");
+  if (ref.namespace !== void 0) validatePart(ref.namespace, "namespace");
+}
+function validateChunkRef(ref) {
+  validateSegmentRef(ref);
+  if (!Number.isInteger(ref.chunkKey) || ref.chunkKey < 0 || ref.chunkKey > CHUNK_KEY_MAX) {
+    throw new ValidationError(
+      `chunkKey must be an integer in [0, ${CHUNK_KEY_MAX}]; got ${ref.chunkKey}`
+    );
+  }
+}
+
+// src/core/concurrency.ts
+async function mapWithConcurrency(items, limit, fn) {
+  if (!Number.isInteger(limit) || limit < 1) {
+    throw new ValidationError(`concurrency limit must be a positive integer; got ${limit}`);
+  }
+  const results = new Array(items.length);
+  let next = 0;
+  let failed = false;
+  let firstError;
+  const worker = async () => {
+    while (!failed) {
+      const i = next++;
+      if (i >= items.length) return;
+      try {
+        results[i] = await fn(items[i], i);
+      } catch (err) {
+        if (!failed) {
+          failed = true;
+          firstError = err;
+        }
+        return;
+      }
+    }
+  };
+  const workers = Array.from({ length: Math.min(limit, items.length) }, () => worker());
+  await Promise.all(workers);
+  if (failed) throw firstError;
+  return results;
+}
+
+// src/drivers/_shared/keys.ts
+var DEFAULT_NAMESPACE = "_default";
+function namespaceKeyPart(namespace) {
+  return namespace === void 0 ? DEFAULT_NAMESPACE : encodeNameForKey(namespace);
+}
+function namespacePathPart(namespace) {
+  return namespace === void 0 ? DEFAULT_NAMESPACE : encodeNameForPath(namespace);
+}
+
+// src/core/ports.ts
+var STORAGE_BACKEND_BRAND = /* @__PURE__ */ Symbol.for("cloudbitmaps.storage-backend");
+function createBackend(halves) {
+  if (halves === null || typeof halves !== "object") {
+    throw new ValidationError("createBackend needs `{ storage, registry }`");
+  }
+  const { storage, registry } = halves;
+  if (storage === null || typeof storage !== "object" || typeof storage.putImmutable !== "function") {
+    throw new ValidationError(
+      "createBackend: `storage` must be an IStorageDriver (it has no `putImmutable`)"
+    );
+  }
+  if (registry === null || typeof registry !== "object" || typeof registry.compareAndSwap !== "function") {
+    throw new ValidationError(
+      "createBackend: `registry` must be an IRegistryDriver (it has no `compareAndSwap`)"
+    );
+  }
+  return brandAsBackend({ storage, registry });
+}
+var STORAGE_BACKEND = STORAGE_BACKEND_BRAND;
+function brandAsBackend(target) {
+  Object.defineProperty(target, STORAGE_BACKEND_BRAND, {
+    value: true,
+    enumerable: false,
+    writable: false,
+    configurable: false
+  });
+  return target;
+}
+function isStorageBackend(value) {
+  return typeof value === "object" && value !== null && value[STORAGE_BACKEND_BRAND] === true;
+}
+
+// src/drivers/_shared/registry.ts
+var STATUSES = ["active", "compacting", "erasing", "destroyed"];
+var MAX_GOVERNANCE_BYTES = 64 * 1024;
+var MAX_WRAPPED_DEKS = 8;
+var MAX_WRAPPED_DEK_BYTES = 4 * 1024;
+function validateGeneration(gen) {
+  if (gen === null) return;
+  if (!Number.isInteger(gen) || gen < 0) {
+    throw new ValidationError(
+      `currentGen must be a non-negative integer or null (no Storage generation yet); got ${gen}`
+    );
+  }
+}
+function validatePatchGeneration(patch) {
+  if (!("currentGen" in patch)) return;
+  if (patch.currentGen === void 0) {
+    throw new ValidationError(
+      `currentGen was present in the patch but undefined. Pass \`null\` to clear the pointer (the segment has no Storage generation), or omit the key to leave it unchanged \u2014 an undefined value is refused because it used to be a no-op and would now un-publish the segment's Storage data.`
+    );
+  }
+  validateGeneration(patch.currentGen);
+}
+function validateStatus(status) {
+  if (!STATUSES.includes(status)) {
+    throw new ValidationError(`status must be one of ${STATUSES.join("/")}; got ${status}`);
+  }
+}
+function validateGovernance(meta, field) {
+  if (meta === void 0) return;
+  if (meta === null || typeof meta !== "object" || Array.isArray(meta)) {
+    throw new ValidationError(
+      `${field} must be a plain object (got ${meta === null ? "null" : typeof meta})`
+    );
+  }
+  let json;
+  try {
+    json = JSON.stringify(meta);
+  } catch {
+    throw new ValidationError(`${field} must be JSON-serializable`);
+  }
+  if (json.length > MAX_GOVERNANCE_BYTES) {
+    throw new ValidationError(`${field} is ${json.length}B, exceeds cap ${MAX_GOVERNANCE_BYTES}B`);
+  }
+}
+function validateWrappedDeks(value, isStored) {
+  if (value === void 0) return;
+  const fail = (msg) => {
+    throw isStored ? new IntegrityError(msg) : new ValidationError(msg);
+  };
+  if (!Array.isArray(value)) fail("wrappedDeks must be an array");
+  const list = value;
+  if (list.length === 0) fail("wrappedDeks must be non-empty when present");
+  if (list.length > MAX_WRAPPED_DEKS)
+    fail(`wrappedDeks has ${list.length} entries, cap ${MAX_WRAPPED_DEKS}`);
+  for (const w of list) {
+    if (w === null || typeof w !== "object") fail("wrappedDeks entry must be an object");
+    const { keyId, wrapped } = w;
+    if (typeof keyId !== "string" || keyId.length === 0)
+      fail("wrappedDeks entry needs a non-empty keyId");
+    if (typeof wrapped !== "string" || wrapped.length === 0)
+      fail("wrappedDeks entry needs a non-empty wrapped blob");
+    if (wrapped.length > MAX_WRAPPED_DEK_BYTES)
+      fail(`wrappedDeks entry is ${wrapped.length}B, cap ${MAX_WRAPPED_DEK_BYTES}B`);
+  }
+}
+function validateNewRegistryRecord(rec) {
+  validateGeneration(rec.currentGen);
+  if (rec.status !== void 0) validateStatus(rec.status);
+  validateWrappedDeks(rec.wrappedDeks, false);
+  validateGovernance(rec.retention, "retention");
+  validateGovernance(rec.residency, "residency");
+}
+function validateRegistryPatch(patch) {
+  validatePatchGeneration(patch);
+  if (patch.status !== void 0) validateStatus(patch.status);
+  if ("wrappedDeks" in patch) validateWrappedDeks(patch.wrappedDeks, false);
+  if ("retention" in patch) validateGovernance(patch.retention, "retention");
+  if ("residency" in patch) validateGovernance(patch.residency, "residency");
+}
+var REGISTRY_SCHEMA_VERSION = 1;
+function assertRegistrySchemaVersion(raw, ctx) {
+  if (raw === void 0) return;
+  if (typeof raw !== "number" || !Number.isInteger(raw) || raw < 1) {
+    throw new IntegrityError(`registry row has a malformed schemaVersion (${String(raw)}): ${ctx}`);
+  }
+  if (raw > REGISTRY_SCHEMA_VERSION) {
+    throw new UnsupportedError(
+      `registry row schemaVersion ${raw} is newer than this build reads (v${REGISTRY_SCHEMA_VERSION}): ${ctx}`
+    );
+  }
+}
+function serializeRegistryEnvelope(env) {
+  return JSON.stringify({
+    schemaVersion: REGISTRY_SCHEMA_VERSION,
+    deleted: env.deleted,
+    record: env.record
+  });
+}
+function registryCounterOf(record) {
+  if (!/^(0|[1-9]\d*)$/.test(record.token)) {
+    throw new IntegrityError(`registry row token is not a canonical counter: ${record.token}`);
+  }
+  const n = Number(record.token);
+  if (!Number.isSafeInteger(n)) {
+    throw new IntegrityError(`registry row token is out of safe-integer range: ${record.token}`);
+  }
+  return n;
+}
+function parseRegistryEnvelope(text, ctx) {
+  let parsed;
+  try {
+    parsed = JSON.parse(text);
+  } catch {
+    throw new IntegrityError(`registry row is not valid JSON: ${ctx}`);
+  }
+  if (parsed === null || typeof parsed !== "object") {
+    throw new IntegrityError(`registry row has a malformed envelope: ${ctx}`);
+  }
+  const env = parsed;
+  assertRegistrySchemaVersion(env.schemaVersion, ctx);
+  if (typeof env.deleted !== "boolean" || env.record === null || typeof env.record !== "object") {
+    throw new IntegrityError(`registry row has a malformed envelope: ${ctx}`);
+  }
+  const r = env.record;
+  assertStoredRecordShape(r, ctx);
+  if (typeof r.token !== "string") {
+    throw new IntegrityError(`registry row is missing its token: ${ctx}`);
+  }
+  return { deleted: env.deleted, record: env.record };
+}
+function recordFromNew(ref, rec, now, token) {
+  return {
+    namespace: ref.namespace,
+    segment: ref.segment,
+    currentGen: rec.currentGen,
+    wrappedDeks: rec.wrappedDeks,
+    keyId: rec.keyId,
+    status: rec.status ?? "active",
+    retention: rec.retention,
+    residency: rec.residency,
+    createdAt: now,
+    updatedAt: now,
+    token
+  };
+}
+function assertStoredRecordShape(r, ctx) {
+  if (typeof r.segment !== "string" || r.namespace !== void 0 && typeof r.namespace !== "string" || r.currentGen !== null && typeof r.currentGen !== "number" || typeof r.status !== "string" || typeof r.createdAt !== "number" || typeof r.updatedAt !== "number") {
+    throw new IntegrityError(`registry record is missing required fields: ${ctx}`);
+  }
+  if (r.currentGen !== null && (!Number.isInteger(r.currentGen) || r.currentGen < 0)) {
+    throw new IntegrityError(`registry record has an invalid currentGen (${r.currentGen}): ${ctx}`);
+  }
+  if (!STATUSES.includes(r.status)) {
+    throw new IntegrityError(`registry record has an unknown status (${r.status}): ${ctx}`);
+  }
+  if (r.keyId !== void 0 && typeof r.keyId !== "string") {
+    throw new IntegrityError(`registry record has an invalid keyId: ${ctx}`);
+  }
+  validateWrappedDeks(r.wrappedDeks, true);
+  for (const field of ["retention", "residency"]) {
+    const meta = r[field];
+    if (meta !== void 0 && (meta === null || typeof meta !== "object" || Array.isArray(meta))) {
+      throw new IntegrityError(`registry record has a non-object ${field}: ${ctx}`);
+    }
+  }
+}
+function applyRegistryPatch(prev, patch, now, token) {
+  return {
+    namespace: prev.namespace,
+    segment: prev.segment,
+    // `'currentGen' in patch`, NOT `patch.currentGen ?? prev.currentGen`. `null` is a legal, meaningful value
+    // here (no Storage generation yet), and `??` treats it as absent — so the nullish form would silently ignore a
+    // patch that clears the pointer and leave the old generation in place. The same reason `wrappedDeks` and
+    // `keyId` below use presence: any field whose null is a *value* cannot be merged with `??`.
+    currentGen: "currentGen" in patch ? patch.currentGen ?? null : prev.currentGen,
+    wrappedDeks: "wrappedDeks" in patch ? patch.wrappedDeks : prev.wrappedDeks,
+    keyId: "keyId" in patch ? patch.keyId : prev.keyId,
+    status: patch.status ?? prev.status,
+    retention: "retention" in patch ? patch.retention : prev.retention,
+    residency: "residency" in patch ? patch.residency : prev.residency,
+    createdAt: prev.createdAt,
+    updatedAt: now,
+    token
+  };
+}
+
+export {
+  CloudRoaringError,
+  ValidationError,
+  WriteConflictError,
+  IntegrityError,
+  NotFoundError,
+  UnsupportedError,
+  CapabilityError,
+  BudgetExceededError,
+  KeyUnavailableError,
+  TransientError,
+  TimeoutError,
+  isCloudRoaringError,
+  isWriteConflictError,
+  isTransientError,
+  isNotFoundError,
+  isIntegrityError,
+  isValidationError,
+  encodeNameForKey,
+  decodeNameFromKey,
+  encodeNameForPath,
+  decodeNameFromPath,
+  MAX_NAME_LENGTH,
+  encodedNameLength,
+  validateSegmentRef,
+  validateChunkRef,
+  mapWithConcurrency,
+  DEFAULT_NAMESPACE,
+  namespaceKeyPart,
+  namespacePathPart,
+  createBackend,
+  STORAGE_BACKEND,
+  brandAsBackend,
+  isStorageBackend,
+  validateNewRegistryRecord,
+  validateRegistryPatch,
+  serializeRegistryEnvelope,
+  registryCounterOf,
+  parseRegistryEnvelope,
+  recordFromNew,
+  applyRegistryPatch
+};
+//# sourceMappingURL=chunk-BUPDGZRR.js.map