rida-format 0.0.0-stage → 0.3.3

package/LICENSE

@@ -0,0 +1,51 @@
+RIDA PROPRIETARY LICENSE
+Version 1.0
+
+Copyright (c) 2026 Syed Ali Hamza Chishty. All rights reserved.
+
+This license applies to the RIDA software library and its source code
+(“RIDA”). It does not apply to data files created with RIDA.
+
+1. Permission to use and modify
+
+You may download, install, run, inspect, and modify RIDA for your own use.
+You may use RIDA, including a modified version, as a component of software
+applications that you create.
+
+2. Applications and commercial use
+
+You may distribute and sell applications that include RIDA, including a
+modified version, as an integrated component. You may not offer RIDA as a
+separately installable or downloadable library, package, or standalone
+software product as part of that distribution.
+
+Every application that includes RIDA must display this notice in an About,
+Licenses, or Third-Party Notices section:
+
+“Uses RIDA (Range Indexed Data Access).
+Copyright (c) 2026 Syed Ali Hamza Chishty.
+Used under the RIDA Proprietary License.”
+
+3. Standalone redistribution and resale
+
+You may not sell, sublicense, publish, or otherwise distribute RIDA or a
+modified version of RIDA as a standalone library, package, or software
+product, except for the Node.js edition in this package. The copyright owner
+specifically permits this Node.js edition to be published and distributed as
+the standalone npm package named “rida-format”, including through the public
+npm registry. Recipients may download, install, run, inspect, and modify this
+Node.js edition for their own use, and may use it as an integrated part of an
+application under Section 2. Recipients may not republish or sell it as a
+standalone package. This exception does not authorize standalone distribution
+of another RIDA edition or under another package name. You may not claim that
+you own or authored the original RIDA code.
+
+4. Ownership
+
+Syed Ali Hamza Chishty retains ownership of RIDA. This license grants only
+the permissions stated above; it does not transfer ownership of RIDA.
+
+5. Warranty
+
+RIDA is provided “AS IS”, without warranties or conditions of any kind,
+express or implied, to the extent permitted by applicable law.

package/README.md

@@ -1,3 +1,84 @@
-# Temporary Holding Version
+# RIDA Node.js tools — 0.3.3
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+This folder contains a Node.js command-line reader and builder for RIDA 0.3.3 files. It reads the same data and index format as the Python release. It does not convert or rewrite an existing dataset when reading it.
+
+## Install for local use
+
+From this folder:
+
+```sh
+npm install
+node ./src/cli.js --help
+```
+
+Node.js 20 or newer is required. `lzma-native` reads and writes the XZ catalog. It includes native code; installation uses a prebuilt binary when available and may need a compiler on other systems.
+
+## Build a dataset
+
+A folder can contain one JSON object per `.json` file. Nested folders are included. By default RIDA gives each folder and record a numbered ID and writes a `Map.json` when those IDs need folder names.
+
+```sh
+node ./src/cli.js scan ./records
+node ./src/cli.js build ./records --output ./books.rida
+```
+
+The build creates `books.rida` and `booksIndex.rida`. It also creates `booksMap.json` when folder IDs need a map. Existing data or index files are not overwritten.
+
+The tool also accepts one large JSON file whose top-level object maps IDs to record objects. A top-level JSON array needs `--id-key`. It reads one record at a time, rather than loading the whole JSON file:
+
+```sh
+node ./src/cli.js build ./records.json --id-key id --output ./records.rida
+```
+
+Useful build options:
+
+```text
+--mode auto|dense|sparse       Choose the catalog layout (default: auto)
+--codec deflate-per-field|plain  Compress each value, or store raw JSON
+--compression 0-9              Set XZ catalog compression (default: 9)
+--encrypt                     Create a fresh AES-256-GCM key
+--key FILE_OR_TEXT             Reuse a 32-byte key in base64url or hex form
+--records-dir FOLDER            Build from a selected folder inside the source
+--combine-groups               Combine immediate JSON groups into one dataset
+--combined-schema common|union Keep only shared fields, or every field (default)
+--named-ids                    Use JSON file names for record IDs
+```
+
+Keep the generated `Key.rida` private. Without it encrypted values cannot be read. The key is separate from the data, index, and map files.
+
+## Read a dataset
+
+```sh
+node ./src/cli.js info ./books.rida
+node ./src/cli.js get ./books.rida 1/1/1/1 translation.urdu --key ./booksKey.rida
+node ./src/cli.js get-many ./books.rida 1/1/1/1 title translation.urdu --key ./booksKey.rida
+node ./src/cli.js record ./books.rida 1/1/1/1 --key ./booksKey.rida
+node ./src/cli.js verify ./books.rida --key ./booksKey.rida
+```
+
+The JavaScript API is also available from `src/index.js`:
+
+```js
+import { RidaReader } from "./src/index.js";
+
+const reader = await RidaReader.open("./books.rida", { key: "base64url-key-text" });
+const value = await reader.get("1/1/1/1", "translation.urdu");
+await reader.close();
+```
+
+It supports local data files and HTTP servers/CDNs that honor byte-range requests. `getMany()` can combine nearby value ranges into fewer requests. The index and optional map stay with the reader; the CDN serves the data file.
+
+## Current boundaries
+
+- Folder builds handle one JSON object per record file. The selected folder is scanned to discover fields, then read again to write values. Large datasets do not need all value contents held in memory at once; record IDs and catalog row metadata are kept in memory.
+- Both folder input and a single large JSON file are read record by record. The IDs and catalog row information stay in memory; the full set of JSON values does not.
+- Multiple JSON groups require `--records-dir` to select one, or `--combine-groups`. Combined builds support both common and union schemas.
+- The CLI provides `scan`, `build`, `info`, `get`, `get-many`, `record`, and `verify`. Python's inspection helpers and automatic remote index-cache installer are not ported yet.
+
+## License and publishing
+
+The included license specifically permits publishing this Node.js edition as the standalone npm package `rida-format`. It does not grant permission to publish other RIDA editions as standalone packages. Keep the included `LICENSE` file with the package.
+
+## Format compatibility
+
+RIDA 0.3.3 stores only value bytes in the data file. The index has a 72-byte header followed by an XZ/LZMA catalog. Catalog rows point to values encoded as compact UTF-8 JSON, either plain or raw Deflate per field. Encryption wraps each encoded value as a 12-byte nonce, AES-256-GCM ciphertext, and 16-byte authentication tag. The map contains numbered folder names and field labels; the key is separate and private.

package/package.json

@@ -1,6 +1,24 @@
 {
   "name": "rida-format",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
-}
+  "version": "0.3.3",
+  "description": "Node.js reader and tools for RIDA 0.3.3 datasets",
+  "type": "module",
+  "engines": { "node": ">=20" },
+  "bin": {
+    "rida": "./src/cli.js",
+    "rida-format": "./src/cli.js"
+  },
+  "exports": {
+    ".": "./src/index.js",
+    "./builder": "./src/builder.js"
+  },
+  "files": ["src/", "README.md", "LICENSE"],
+  "scripts": {
+    "start": "node ./src/cli.js",
+    "check": "node --check src/index.js && node --check src/builder.js && node --check src/cli.js"
+  },
+  "dependencies": {
+    "lzma-native": "^8.0.6"
+  },
+  "license": "SEE LICENSE IN LICENSE"
+}

package/src/builder.js

@@ -0,0 +1,520 @@
+import { createHash, randomBytes, createCipheriv } from "node:crypto";
+import { readdir, readFile, mkdir, open, rename, rm, stat, writeFile } from "node:fs/promises";
+import { createReadStream } from "node:fs";
+import { basename, dirname, extname, join, relative, resolve } from "node:path";
+import { deflateRawSync } from "node:zlib";
+import { createRequire } from "node:module";
+import { RidaError } from "./index.js";
+
+const require = createRequire(import.meta.url);
+const lzma = require("lzma-native");
+const INDEX_MAGIC = Buffer.from("RIDAIDX1");
+const CATALOG_MAGIC = Buffer.from("RDACAT01");
+const HEADER_SIZE = 72;
+
+export function compactJSON(value) {
+  const text = JSON.stringify(value, (_, item) => {
+    if (typeof item === "number" && !Number.isFinite(item)) throw new RidaError("RIDA stores standard JSON only; NaN and Infinity are not supported");
+    return item;
+  });
+  if (text === undefined) throw new RidaError("Value cannot be represented as JSON");
+  return Buffer.from(text, "utf8");
+}
+
+export function encodeVarint(input) {
+  let value = BigInt(input);
+  const bytes = [];
+  while (value >= 128n) { bytes.push(Number(value & 127n) | 128); value >>= 7n; }
+  bytes.push(Number(value));
+  return Buffer.from(bytes);
+}
+
+function orderKey(name) {
+  const stem = name.slice(0, name.lastIndexOf("."));
+  return /^\d+$/.test(stem) ? [0, BigInt(stem), name.toLowerCase()] : [1, stem.toLowerCase(), name.toLowerCase()];
+}
+const lexical = (a, b) => a < b ? -1 : a > b ? 1 : 0;
+function compareNames(a, b) {
+  const x = orderKey(a), y = orderKey(b);
+  if (x[0] !== y[0]) return x[0] - y[0];
+  if (x[1] !== y[1]) return x[1] < y[1] ? -1 : 1;
+  return lexical(x[2], y[2]);
+}
+
+async function walkJSON(root, { recursive = true } = {}) {
+  const files = [];
+  async function walk(dir) {
+    const entries = (await readdir(dir, { withFileTypes: true })).filter((entry) => !entry.name.startsWith(".")).sort((a, b) => lexical(a.name, b.name));
+    for (const entry of entries) {
+      const path = join(dir, entry.name);
+      if (entry.isDirectory()) { if (recursive) await walk(path); continue; }
+      if (entry.isFile() && /\.json$/i.test(entry.name)) files.push(path);
+    }
+  }
+  await walk(root);
+  files.sort((a, b) => lexical(relative(root, a).split(/[\\/]/).join("/"), relative(root, b).split(/[\\/]/).join("/")));
+  if (!files.length) throw new RidaError(`No JSON files found in ${root}`);
+  return files;
+}
+
+function numberedIds(paths, root) {
+  const folderChildren = new Map();
+  const filesByFolder = new Map();
+  const rootText = resolve(root);
+  for (const path of paths) {
+    const folder = dirname(path);
+    if (!filesByFolder.has(folder)) filesByFolder.set(folder, []);
+    filesByFolder.get(folder).push(path);
+    let current = folder;
+    while (current !== rootText && current.startsWith(rootText + "/")) {
+      const parent = dirname(current);
+      if (!folderChildren.has(parent)) folderChildren.set(parent, new Set());
+      folderChildren.get(parent).add(current);
+      current = parent;
+    }
+  }
+  const folderIds = new Map([[rootText, ""]]);
+  const queue = [rootText];
+  const mapPaths = {};
+  while (queue.length) {
+    const parent = queue.shift();
+    const children = [...(folderChildren.get(parent) ?? [])].sort((a, b) => lexical(basename(a), basename(b)));
+    for (let i = 0; i < children.length; i++) {
+      const child = children[i];
+      const prefix = [folderIds.get(parent), String(i + 1)].filter(Boolean).join("/");
+      folderIds.set(child, prefix);
+      mapPaths[prefix] = basename(child);
+      queue.push(child);
+    }
+  }
+  const ids = new Map();
+  for (const [folder, folderFiles] of filesByFolder) {
+    folderFiles.sort((a, b) => compareNames(basename(a), basename(b)));
+    const prefix = folderIds.get(folder) ?? "";
+    folderFiles.forEach((path, i) => ids.set(path, [prefix, String(i + 1)].filter(Boolean).join("/")));
+  }
+  return { ids, paths: mapPaths };
+}
+
+function uniqueId(id, used) {
+  if (!used.has(id)) { used.add(id); return id; }
+  let n = 2;
+  while (used.has(`${id}-${n}`)) n++;
+  const unique = `${id}-${n}`;
+  used.add(unique);
+  return unique;
+}
+
+async function readRecordFile(path) {
+  let value;
+  try { value = JSON.parse(await readFile(path, "utf8")); }
+  catch (cause) { throw new RidaError(`Invalid JSON in ${path}: ${cause.message}`, { cause }); }
+  if (!value || typeof value !== "object" || Array.isArray(value)) throw new RidaError(`Record file must contain one JSON object: ${path}`);
+  return value;
+}
+
+class JsonCursor {
+  constructor(path) {
+    this.chunks = createReadStream(path, { encoding: "utf8" })[Symbol.asyncIterator]();
+    this.chunk = "";
+    this.position = 0;
+    this.ended = false;
+    this.started = false;
+    this.path = path;
+  }
+  async _fill() {
+    while (this.position >= this.chunk.length && !this.ended) {
+      const next = await this.chunks.next();
+      if (next.done) { this.ended = true; this.chunk = ""; this.position = 0; }
+      else { this.chunk = next.value; this.position = 0; }
+    }
+  }
+  async peek() {
+    await this._fill();
+    if (!this.started && !this.ended) {
+      this.started = true;
+      if (this.chunk[this.position] === "\uFEFF") this.position++;
+      await this._fill();
+    }
+    return this.ended ? null : this.chunk[this.position];
+  }
+  async take() { const value = await this.peek(); if (value !== null) this.position++; return value; }
+  async skipWhitespace() { while (/\s/.test(await this.peek() ?? "")) await this.take(); }
+  async expect(expected) {
+    await this.skipWhitespace();
+    const actual = await this.take();
+    if (actual !== expected) throw new RidaError(`Invalid JSON in ${this.path}: expected ${JSON.stringify(expected)}, found ${JSON.stringify(actual)}`);
+  }
+  async readValueText() {
+    await this.skipWhitespace();
+    const first = await this.peek();
+    if (first === null) throw new RidaError(`Invalid or incomplete JSON in ${this.path}`);
+    const out = [];
+    if (first === '"') {
+      let escaped = false;
+      while (true) {
+        const char = await this.take();
+        if (char === null) throw new RidaError(`Invalid or incomplete JSON string in ${this.path}`);
+        out.push(char);
+        if (escaped) escaped = false;
+        else if (char === "\\") escaped = true;
+        else if (char === '"') break;
+      }
+      return out.join("");
+    }
+    if (first === "{" || first === "[") {
+      const stack = [];
+      let inString = false;
+      let escaped = false;
+      while (true) {
+        const char = await this.take();
+        if (char === null) throw new RidaError(`Invalid or incomplete JSON value in ${this.path}`);
+        out.push(char);
+        if (inString) {
+          if (escaped) escaped = false;
+          else if (char === "\\") escaped = true;
+          else if (char === '"') inString = false;
+          continue;
+        }
+        if (char === '"') inString = true;
+        else if (char === "{" || char === "[") stack.push(char);
+        else if (char === "}" || char === "]") {
+          const opening = stack.pop();
+          if ((opening === "{" && char !== "}") || (opening === "[" && char !== "]")) throw new RidaError(`Mismatched JSON delimiters in ${this.path}`);
+          if (!stack.length) break;
+        }
+      }
+      return out.join("");
+    }
+    while (true) {
+      const char = await this.peek();
+      if (char === null || /[\s,\]}]/.test(char)) break;
+      out.push(await this.take());
+    }
+    return out.join("");
+  }
+  async finish() {
+    await this.skipWhitespace();
+    if (await this.peek() !== null) throw new RidaError(`Unexpected content after JSON value in ${this.path}`);
+  }
+}
+
+async function* jsonRecords(path, idKey) {
+  const cursor = new JsonCursor(path);
+  await cursor.skipWhitespace();
+  const top = await cursor.peek();
+  if (top === "{") {
+    await cursor.expect("{");
+    if (await cursor.peek() === "}") { await cursor.take(); await cursor.finish(); return; }
+    while (true) {
+      const idText = await cursor.readValueText();
+      let id;
+      try { id = JSON.parse(idText); } catch { throw new RidaError(`Invalid record ID in ${path}`); }
+      if (typeof id !== "string") throw new RidaError(`JSON object record IDs must be strings in ${path}`);
+      await cursor.expect(":");
+      const recordText = await cursor.readValueText();
+      let record;
+      try { record = JSON.parse(recordText); } catch (cause) { throw new RidaError(`Invalid JSON record ${id} in ${path}: ${cause.message}`, { cause }); }
+      if (!record || typeof record !== "object" || Array.isArray(record)) throw new RidaError(`Record ${id} is not a JSON object`);
+      yield [id, record];
+      await cursor.skipWhitespace();
+      const delimiter = await cursor.take();
+      if (delimiter === "}") break;
+      if (delimiter !== ",") throw new RidaError(`Invalid JSON object in ${path}: expected ',' or '}'`);
+    }
+    await cursor.finish();
+    return;
+  }
+  if (top === "[") {
+    if (!idKey) throw new RidaError("A JSON array needs --id-key so RIDA can identify every record");
+    await cursor.expect("[");
+    if (await cursor.peek() === "]") { await cursor.take(); await cursor.finish(); return; }
+    let count = 0;
+    while (true) {
+      const recordText = await cursor.readValueText();
+      let record;
+      try { record = JSON.parse(recordText); } catch (cause) { throw new RidaError(`Invalid JSON array item in ${path}: ${cause.message}`, { cause }); }
+      count++;
+      if (!record || typeof record !== "object" || Array.isArray(record)) throw new RidaError(`Array item ${count} is not a JSON object`);
+      if (!Object.hasOwn(record, idKey) || record[idKey] == null) throw new RidaError(`Array item ${count} has no usable ${idKey} ID`);
+      yield [String(record[idKey]), record];
+      await cursor.skipWhitespace();
+      const delimiter = await cursor.take();
+      if (delimiter === "]") break;
+      if (delimiter !== ",") throw new RidaError(`Invalid JSON array in ${path}: expected ',' or ']'`);
+    }
+    await cursor.finish();
+    return;
+  }
+  throw new RidaError(`JSON input must be an object mapping IDs to records or an array: ${path}`);
+}
+
+function encodeValue(value, codec, key) {
+  let bytes = compactJSON(value);
+  if (codec === "deflate-per-field") bytes = deflateRawSync(bytes, { level: 9 });
+  if (key) {
+    const nonce = randomBytes(12);
+    const cipher = createCipheriv("aes-256-gcm", key, nonce);
+    bytes = Buffer.concat([nonce, cipher.update(bytes), cipher.final(), cipher.getAuthTag()]);
+  }
+  return bytes;
+}
+
+function keyBytes(value) {
+  if (!value) return null;
+  if (Buffer.isBuffer(value)) return value;
+  const text = String(value).trim();
+  const decoded = Buffer.from(text, "base64url");
+  if (decoded.length === 32) return decoded;
+  const hex = Buffer.from(text, "hex");
+  if (hex.length === 32) return hex;
+  throw new RidaError("Encryption key must be 32 bytes, base64, or hex");
+}
+
+function companion(output, suffix) {
+  let stem = basename(output, extname(output));
+  if (stem.endsWith("Data")) stem = stem.slice(0, -4);
+  return join(dirname(output), `${stem}${suffix}`);
+}
+
+function progress(label, done, total) {
+  const width = 24;
+  const ratio = total > 0 ? Math.min(1, done / total) : 0;
+  const filled = Math.floor(width * ratio);
+  process.stderr.write(`\r${label}: ${Math.round(ratio * 100)}% [${"█".repeat(filled)}${"░".repeat(width - filled)}] ${done.toLocaleString()}${total ? `/${total.toLocaleString()}` : ""}`);
+  if (total && done >= total) process.stderr.write("\n");
+}
+
+export async function scan(source, options = {}) {
+  let sourcePath = resolve(source);
+  let sourceStat = await stat(sourcePath);
+  let discoveredFiles = null;
+  if (!sourceStat.isDirectory()) {
+    const fields = [];
+    const seenFields = new Set();
+    const usedIds = new Set();
+    const records = [];
+    let count = 0;
+    for await (const [rawId, record] of jsonRecords(sourcePath, options.idKey)) {
+      count++;
+      for (const field of Object.keys(record)) if (!seenFields.has(field)) { seenFields.add(field); fields.push(field); }
+      records.push({ id: uniqueId(String(rawId), usedIds), sourceIndex: count - 1 });
+      if (count % 1000 === 0) progress("Scanning JSON records", count, 0);
+    }
+    if (!count) throw new RidaError("RIDA needs at least one record");
+    if (!fields.length) throw new RidaError("RIDA records must contain at least one field");
+    progress("Scanning JSON records", count, count);
+    return { sourcePath, files: [], records, fields, mapPaths: {}, recordCount: records.length, singleJSON: true, idKey: options.idKey };
+  }
+  if (options.recordsDir) {
+    sourcePath = resolve(sourcePath, options.recordsDir);
+    sourceStat = await stat(sourcePath);
+    if (!sourceStat.isDirectory()) throw new RidaError(`Records folder does not exist: ${sourcePath}`);
+  } else {
+    const entries = await readdir(sourcePath, { withFileTypes: true });
+    const directFiles = entries.filter((entry) => entry.isFile() && /\.json$/i.test(entry.name) && !entry.name.startsWith("."));
+    const directJSON = directFiles.length > 0;
+    const groups = [];
+    if (directJSON) groups.push({ path: sourcePath, files: directFiles.map((entry) => join(sourcePath, entry.name)) });
+    for (const entry of entries) {
+      if (!entry.isDirectory() || entry.name.startsWith(".")) continue;
+      const groupPath = join(sourcePath, entry.name);
+      try { groups.push({ path: groupPath, files: await walkJSON(groupPath) }); } catch (error) { if (!(error instanceof RidaError)) throw error; }
+    }
+    if (groups.length > 1 && !options.combineGroups) {
+      throw new RidaError(`This folder has ${groups.length} JSON groups. Choose one with --records-dir, or use --combine-groups.`);
+    }
+    if (groups.length === 1 && !options.combineGroups) {
+      sourcePath = groups[0].path;
+      discoveredFiles = groups[0].files;
+    } else if (options.combineGroups) {
+      discoveredFiles = groups.flatMap((group) => group.files);
+    }
+  }
+  if (![undefined, "common", "union"].includes(options.combinedSchema)) throw new RidaError("Combined schema must be common or union");
+  const files = discoveredFiles ?? await walkJSON(sourcePath);
+  files.sort((a, b) => lexical(relative(sourcePath, a).split(/[\\/]/).join("/"), relative(sourcePath, b).split(/[\\/]/).join("/")));
+  const mapping = options.numbered === false ? null : numberedIds(files, sourcePath);
+  const records = [];
+  const fields = [];
+  const seenFields = new Set();
+  const usedIds = new Set();
+  for (let i = 0; i < files.length; i++) {
+    let record = await readRecordFile(files[i]);
+    let id = options.numbered === false ? basename(files[i], extname(files[i])) : mapping.ids.get(files[i]);
+    if (options.numbered === false) id = uniqueId(id, usedIds);
+    records.push({ id: options.numbered === false ? id : (usedIds.add(id), id), file: files[i] });
+    for (const field of Object.keys(record)) if (!seenFields.has(field)) { seenFields.add(field); fields.push(field); }
+    if ((i + 1) % 1000 === 0 || i + 1 === files.length) progress("Scanning JSON records", i + 1, files.length);
+  }
+  if (options.combineGroups && options.combinedSchema === "common") {
+    const groupFields = new Map();
+    for (let i = 0; i < files.length; i++) {
+      const relativePath = relative(sourcePath, files[i]);
+      const firstFolder = relativePath.includes("/") ? relativePath.split("/")[0] : ".";
+      if (!groupFields.has(firstFolder)) groupFields.set(firstFolder, new Set());
+      for (const field of Object.keys(await readRecordFile(files[i]))) groupFields.get(firstFolder).add(field);
+    }
+    const [first, ...rest] = [...groupFields.values()];
+    const common = new Set([...first].filter((field) => rest.every((fields) => fields.has(field))));
+    for (let i = fields.length - 1; i >= 0; i--) if (!common.has(fields[i])) fields.splice(i, 1);
+  }
+  if (!fields.length) throw new RidaError("RIDA records must contain at least one field");
+  return { sourcePath, files, records, fields, mapPaths: mapping?.paths ?? {}, recordCount: records.length };
+}
+
+export async function build(source, output, options = {}) {
+  const codec = options.codec ?? "deflate-per-field";
+  if (!["deflate-per-field", "plain"].includes(codec)) throw new RidaError("Codec must be deflate-per-field or plain");
+  if (![undefined, "auto", "dense", "sparse"].includes(options.mode)) throw new RidaError("Mode must be auto, dense, or sparse");
+  if (options.encrypt && options.key) throw new RidaError("Choose either --encrypt for a new key or --key to reuse a key");
+  const compression = Number(options.compression ?? 9);
+  if (!Number.isInteger(compression) || compression < 0 || compression > 9) throw new RidaError("Catalog compression must be from 0 to 9");
+  const key = options.encrypt ? randomBytes(32) : keyBytes(options.key);
+  const plan = await scan(source, options);
+  const target = resolve(output);
+  const plannedOutputs = [target, companion(target, "Index.rida")];
+  if (key) plannedOutputs.push(companion(target, "Key.rida"));
+  if (Object.keys(plan.mapPaths).length) plannedOutputs.push(companion(target, "Map.json"));
+  if (await Promise.all(plannedOutputs.map(async (path) => { try { await stat(path); return path; } catch { return null; } })).then((values) => values.find(Boolean))) {
+    throw new RidaError(`An output file already exists. Choose another output name: ${target}`);
+  }
+  await mkdir(dirname(target), { recursive: true });
+  const tempData = `${target}.data-tmp-${process.pid}`;
+  const tempDense = `${target}.dense-tmp-${process.pid}`;
+  const tempSparse = `${target}.sparse-tmp-${process.pid}`;
+  const tempDenseOffsets = `${target}.dense-offsets-tmp-${process.pid}`;
+  const tempSparseOffsets = `${target}.sparse-offsets-tmp-${process.pid}`;
+  let dataLength = 0;
+  const digest = createHash("sha256");
+  const sourceIterator = plan.singleJSON ? jsonRecords(plan.sourcePath, plan.idKey)[Symbol.asyncIterator]() : null;
+  const temporary = await open(tempData, "wx");
+  const denseFile = await open(tempDense, "wx");
+  const sparseFile = await open(tempSparse, "wx");
+  const denseOffsetsFile = await open(tempDenseOffsets, "wx");
+  const sparseOffsetsFile = await open(tempSparseOffsets, "wx");
+  let denseRowsLength = 0;
+  let sparseRowsLength = 0;
+  let denseOffset = 0;
+  let sparseOffset = 0;
+  try {
+    for (let index = 0; index < plan.records.length; index++) {
+      const { file } = plan.records[index];
+      let record;
+      if (sourceIterator) {
+        const next = await sourceIterator.next();
+        if (next.done) throw new RidaError("Source JSON changed between scanning and building");
+        record = next.value[1];
+      } else record = await readRecordFile(file);
+      const recordDataStart = dataLength;
+      const denseParts = [encodeVarint(recordDataStart)];
+      const presentParts = [];
+      let presentCount = 0;
+      let previousFieldId = 0;
+      for (let fieldId = 0; fieldId < plan.fields.length; fieldId++) {
+        const field = plan.fields[fieldId];
+        if (!Object.hasOwn(record, field)) { denseParts.push(encodeVarint(0)); continue; }
+        const block = encodeValue(record[field], codec, key);
+        denseParts.push(encodeVarint(block.length));
+        presentParts.push(encodeVarint(fieldId - previousFieldId), encodeVarint(block.length));
+        previousFieldId = fieldId;
+        presentCount++;
+        await temporary.write(block);
+        digest.update(block);
+        dataLength += block.length;
+      }
+      const denseRow = Buffer.concat(denseParts);
+      const sparseRow = Buffer.concat([encodeVarint(recordDataStart), encodeVarint(presentCount), ...presentParts]);
+      await denseFile.write(denseRow);
+      await sparseFile.write(sparseRow);
+      denseRowsLength += denseRow.length;
+      sparseRowsLength += sparseRow.length;
+      denseOffset += denseRow.length;
+      sparseOffset += sparseRow.length;
+      const q = Buffer.alloc(8);
+      q.writeBigUInt64BE(BigInt(denseOffset));
+      await denseOffsetsFile.write(q);
+      q.writeBigUInt64BE(BigInt(sparseOffset));
+      await sparseOffsetsFile.write(q);
+      if ((index + 1) % 1000 === 0 || index + 1 === plan.records.length) progress("Writing RIDA values", index + 1, plan.records.length);
+    }
+    if (sourceIterator) {
+      const end = await sourceIterator.next();
+      if (!end.done) throw new RidaError("Source JSON changed between scanning and building");
+    }
+    await temporary.sync();
+    await denseFile.sync();
+    await sparseFile.sync();
+    await denseOffsetsFile.sync();
+    await sparseOffsetsFile.sync();
+  } catch (error) {
+    await sourceIterator?.return?.();
+    await Promise.all([temporary, denseFile, sparseFile, denseOffsetsFile, sparseOffsetsFile].map((file) => file.close().catch(() => {})));
+    await Promise.all([tempData, tempDense, tempSparse, tempDenseOffsets, tempSparseOffsets].map((path) => rm(path, { force: true })));
+    throw error;
+  }
+  await Promise.all([temporary, denseFile, sparseFile, denseOffsetsFile, sparseOffsetsFile].map((file) => file.close()));
+  let mode = options.mode;
+  if (!mode || mode === "auto") {
+    const common = 44 + compactJSON(plan.fields).length + (plan.records.length + 1) * 8;
+    mode = common + denseRowsLength <= common + sparseRowsLength ? "dense" : "sparse";
+  }
+  const sparse = mode === "sparse";
+  const selectedRows = await readFile(sparse ? tempSparse : tempDense);
+  const selectedOffsets = await readFile(sparse ? tempSparseOffsets : tempDenseOffsets);
+  const rowOffsets = [0];
+  for (let offset = 0; offset < selectedOffsets.length; offset += 8) rowOffsets.push(Number(selectedOffsets.readBigUInt64BE(offset)));
+  const chosenLength = sparse ? sparseRowsLength : denseRowsLength;
+  const ids = plan.records.map((record) => record.id);
+  const sequential = ids.every((id, index) => id === String(index + 1));
+  const schema = compactJSON(plan.fields);
+  const idBytes = sequential ? Buffer.alloc(0) : compactJSON(ids);
+  const offsets = Buffer.alloc(rowOffsets.length * 8);
+  rowOffsets.forEach((offset, index) => offsets.writeBigUInt64BE(BigInt(offset), index * 8));
+  const catalogHeader = Buffer.alloc(44);
+  CATALOG_MAGIC.copy(catalogHeader, 0);
+  catalogHeader.writeUInt32BE(1, 8);
+  catalogHeader.writeUInt32BE(sparse ? 2 : 1, 12);
+  catalogHeader.writeBigUInt64BE(BigInt(plan.records.length), 16);
+  catalogHeader.writeUInt32BE(plan.fields.length, 24);
+  catalogHeader.writeUInt32BE(schema.length, 28);
+  catalogHeader.writeUInt32BE(idBytes.length, 32);
+  catalogHeader.writeBigUInt64BE(BigInt(chosenLength), 36);
+  const rawCatalog = Buffer.concat([catalogHeader, schema, idBytes, offsets, selectedRows]);
+  const compressedCatalog = Buffer.from(await lzma.compress(rawCatalog, { preset: compression }));
+  const indexHeader = Buffer.alloc(HEADER_SIZE);
+  INDEX_MAGIC.copy(indexHeader, 0);
+  indexHeader.writeUInt32BE(1, 8);
+  indexHeader.writeUInt32BE(compressedCatalog.length, 12);
+  indexHeader.writeUInt32BE(rawCatalog.length, 16);
+  indexHeader.writeUInt32BE(plan.records.length, 20);
+  indexHeader.writeBigUInt64BE(BigInt(dataLength), 24);
+  indexHeader.writeUInt32BE(codec === "plain" ? 2 : 1, 32);
+  indexHeader.writeUInt32BE(key ? 1 : 0, 36);
+  digest.digest().copy(indexHeader, 40);
+  const indexBytes = Buffer.concat([indexHeader, compressedCatalog]);
+  const indexPath = companion(target, "Index.rida");
+  const mapPath = companion(target, "Map.json");
+  try {
+    await rename(tempData, target);
+    await writeFile(indexPath, indexBytes, { flag: "wx" });
+    if (key) {
+      const keyText = `${key.toString("base64url")}=`;
+      await writeFile(companion(target, "Key.rida"), `${keyText}\n`, { flag: "wx", mode: 0o600 });
+    }
+    if (Object.keys(plan.mapPaths).length) {
+      const fieldMap = Object.fromEntries(plan.fields.map((field, i) => [String(i + 1), field]));
+      await writeFile(mapPath, `${JSON.stringify({ version: 1, records: plan.recordCount, paths: plan.mapPaths, fields: fieldMap })}\n`, { flag: "wx" });
+    }
+  } catch (error) {
+    await rm(tempData, { force: true });
+    throw error;
+  }
+  await Promise.all([tempDense, tempSparse, tempDenseOffsets, tempSparseOffsets].map((path) => rm(path, { force: true })));
+  return { output: target, index: indexPath, map: Object.keys(plan.mapPaths).length ? mapPath : null, records: plan.recordCount, fields: plan.fields.length, mode, codec, encrypted: Boolean(key), dataBytes: dataLength, catalogBytes: compressedCatalog.length };
+}
+
+export async function inspectSource(source) {
+  const info = await scan(source);
+  return { records: info.recordCount, fields: info.fields, groups: info.mapPaths };
+}

package/src/cli.js

@@ -0,0 +1,100 @@
+#!/usr/bin/env node
+import { readFile } from "node:fs/promises";
+import { basename, resolve } from "node:path";
+import { RidaReader, version } from "./index.js";
+import { build, scan } from "./builder.js";
+
+const HELP = `RIDA Node.js tools ${version}
+
+Usage:
+  rida scan <json-file-or-folder> [--records-dir FOLDER] [--id-key FIELD] [--named-ids]
+  rida build <json-file-or-folder> [--output FILE] [--mode auto|dense|sparse]
+             [--codec deflate-per-field|plain] [--compression 0-9]
+             [--encrypt | --key KEY_OR_FILE] [--records-dir FOLDER]
+             [--combine-groups] [--combined-schema common|union]
+             [--id-key FIELD] [--named-ids]
+  rida info <data.rida> [--index FILE] [--map FILE]
+  rida get <data.rida|URL> <record-id> <field> [--index FILE] [--map FILE] [--key KEY_OR_FILE]
+  rida get-many <data.rida|URL> <record-id> <field>... [--index FILE] [--map FILE] [--key KEY_OR_FILE]
+  rida record <data.rida|URL> <record-id> [--index FILE] [--map FILE] [--key KEY_OR_FILE]
+  rida verify <data.rida|URL> [--index FILE] [--map FILE] [--key KEY_OR_FILE]
+
+Folder input: each visible .json file is one record object. Subfolders are
+numbered into stable record IDs and a companion map is written when needed.
+Single JSON input: an object maps record IDs to records; an array needs --id-key.
+Build writes the data, Index.rida, optional Map.json, and optional Key.rida.
+Existing output files are never overwritten.
+`;
+
+const VALUE_OPTIONS = new Set(["--output", "--mode", "--codec", "--compression", "--key", "--id-key", "--records-dir", "--index", "--map"]);
+const FLAGS = new Set(["--encrypt", "--named-ids", "--combine-groups", "--help", "-h"]);
+function parseArgs(args) {
+  const positional = [];
+  const options = {};
+  for (let i = 0; i < args.length; i++) {
+    const arg = args[i];
+    if (FLAGS.has(arg)) options[arg] = true;
+    else if (VALUE_OPTIONS.has(arg)) {
+      const value = args[++i];
+      if (!value || value.startsWith("--")) throw new Error(`Missing value after ${arg}`);
+      options[arg.slice(2).replaceAll("-", "")] = value;
+    } else if (arg.startsWith("--")) throw new Error(`Unknown option: ${arg}`);
+    else positional.push(arg);
+  }
+  return { positional, options };
+}
+async function resolveKey(key) {
+  if (!key) return undefined;
+  try { return await readFile(key, "utf8"); }
+  catch { return key; }
+}
+function output(value) { process.stdout.write(`${JSON.stringify(value, null, 2)}\n`); }
+
+async function main() {
+  const args = process.argv.slice(2);
+  if (!args.length || ["--help", "-h", "help"].includes(args[0])) { process.stdout.write(HELP); return; }
+  if (["--version", "-v", "version"].includes(args[0])) { process.stdout.write(`${version}\n`); return; }
+  const command = args.shift();
+  const { positional, options } = parseArgs(args);
+  if (command === "scan") {
+    if (positional.length !== 1) throw new Error(`Usage: rida scan <json-file-or-folder>\n\n${HELP}`);
+    const result = await scan(positional[0], { idKey: options.idkey, recordsDir: options.recordsdir, numbered: !options["--named-ids"], combineGroups: options["--combine-groups"], combinedSchema: options.combinedschema ?? "union" });
+    output({ records: result.recordCount, fields: result.fields.length, fieldNames: result.fields, mappedFolders: Object.keys(result.mapPaths).length });
+    return;
+  }
+  if (command === "build") {
+    if (positional.length !== 1) throw new Error(`Usage: rida build <json-file-or-folder> [options]\n\n${HELP}`);
+    const source = positional[0];
+    const outputPath = options.output ?? `${basename(resolve(source)).replace(/\.json$/i, "")}.rida`;
+    const result = await build(source, outputPath, {
+      idKey: options.idkey,
+      recordsDir: options.recordsdir,
+      combineGroups: Boolean(options["--combine-groups"]),
+      combinedSchema: options.combinedschema ?? "union",
+      numbered: !options["--named-ids"],
+      mode: options.mode ?? "auto",
+      codec: options.codec ?? "deflate-per-field",
+      compression: options.compression ?? 9,
+      encrypt: Boolean(options["--encrypt"]),
+      key: await resolveKey(options.key),
+    });
+    output(result);
+    return;
+  }
+  const minimum = { info: 1, get: 3, "get-many": 3, record: 2, verify: 1 }[command];
+  if (!minimum || positional.length < minimum) throw new Error(`Invalid ${command ?? ""} command.\n\n${HELP}`);
+  const [data, recordId, ...fields] = positional;
+  const reader = await RidaReader.open(data, { index: options.index, map: options.map, key: await resolveKey(options.key) });
+  try {
+    if (command === "info") output({ format: "RIDA", version, records: reader.recordCount, fields: reader.fields.length, valueCodec: reader.valueCodec, indexMode: reader.mode, encrypted: reader.info.encrypted, dataBytes: reader.info.dataLength, fieldNames: reader.fields });
+    else if (command === "get") output(await reader.get(recordId, fields[0]));
+    else if (command === "get-many") output(await reader.getMany(recordId, fields));
+    else if (command === "record") output(await reader.getRecord(recordId));
+    else output({ valid: await reader.verifyData(), dataBytes: reader.info.dataLength });
+  } finally { await reader.close(); }
+}
+
+main().catch((error) => {
+  process.stderr.write(`rida: ${error.message}\n`);
+  process.exitCode = 1;
+});

package/src/index.js

@@ -0,0 +1,360 @@
+import { createDecipheriv, createHash } from "node:crypto";
+import { open, readFile } from "node:fs/promises";
+import { basename, dirname, extname, join, parse } from "node:path";
+import { createRequire } from "node:module";
+import { inflateRawSync } from "node:zlib";
+
+const require = createRequire(import.meta.url);
+const lzma = require("lzma-native");
+
+export const version = "0.3.3";
+export const MERGE_GAP_LIMIT = 660;
+const INDEX_MAGIC = Buffer.from("RIDAIDX1");
+const CATALOG_MAGIC = Buffer.from("RDACAT01");
+
+export class RidaError extends Error {
+  constructor(message, options) { super(message, options); this.name = "RidaError"; }
+}
+
+function need(condition, message) {
+  if (!condition) throw new RidaError(message);
+}
+
+function readVarint(bytes, start) {
+  let value = 0n;
+  let shift = 0n;
+  let pos = start;
+  while (pos < bytes.length && shift <= 63n) {
+    const byte = bytes[pos++];
+    value |= BigInt(byte & 0x7f) << shift;
+    if (!(byte & 0x80)) {
+      need(value <= BigInt(Number.MAX_SAFE_INTEGER), "RIDA catalog offset is too large for this Node.js runtime");
+      return [Number(value), pos];
+    }
+    shift += 7n;
+  }
+  throw new RidaError("Invalid variable-length value in RIDA catalog");
+}
+
+function parseIndex(raw) {
+  need(raw.length >= 72, "RIDA index is incomplete");
+  need(raw.subarray(0, 8).equals(INDEX_MAGIC), "Unsupported or invalid RIDA index");
+  const versionNo = raw.readUInt32BE(8);
+  const catalogLength = raw.readUInt32BE(12);
+  const rawCatalogLength = raw.readUInt32BE(16);
+  const recordCount = raw.readUInt32BE(20);
+  const dataLengthBig = raw.readBigUInt64BE(24);
+  need(versionNo === 1, `Unsupported RIDA index version: ${versionNo}`);
+  need(dataLengthBig <= BigInt(Number.MAX_SAFE_INTEGER), "RIDA data file is too large for this Node.js runtime");
+  const codecNo = raw.readUInt32BE(32);
+  need(codecNo === 1 || codecNo === 2, "RIDA index has an unknown value codec");
+  const catalog = raw.subarray(72);
+  need(catalog.length === catalogLength, "RIDA catalog is incomplete");
+  return {
+    info: {
+      version: 3,
+      indexVersion: versionNo,
+      recordCount,
+      dataLength: Number(dataLengthBig),
+      valueCodec: codecNo === 1 ? "deflate-per-field" : "plain",
+      encrypted: Boolean(raw.readUInt32BE(36) & 1),
+      digest: Buffer.from(raw.subarray(40, 72)),
+      catalogLength,
+      rawCatalogLength,
+    },
+    catalog,
+  };
+}
+
+async function parseCatalog(compressed, info) {
+  let raw;
+  try { raw = await lzma.decompress(compressed); }
+  catch (cause) { throw new RidaError("RIDA catalog cannot be decompressed", { cause }); }
+  raw = Buffer.from(raw);
+  need(raw.length === info.rawCatalogLength, "RIDA catalog length does not match its index");
+  need(raw.length >= 44 && raw.subarray(0, 8).equals(CATALOG_MAGIC), "Unsupported or invalid RIDA catalog");
+  need(raw.readUInt32BE(8) === 1, "Unsupported RIDA catalog version");
+  const modeCode = raw.readUInt32BE(12);
+  need(modeCode === 1 || modeCode === 2, "RIDA catalog has an invalid index mode");
+  const recordCountBig = raw.readBigUInt64BE(16);
+  need(recordCountBig === BigInt(info.recordCount), "RIDA record count does not match its catalog");
+  const fieldCount = raw.readUInt32BE(24);
+  const schemaLength = raw.readUInt32BE(28);
+  const idsLength = raw.readUInt32BE(32);
+  const rowsLengthBig = raw.readBigUInt64BE(36);
+  need(rowsLengthBig <= BigInt(Number.MAX_SAFE_INTEGER), "RIDA catalog is too large for this Node.js runtime");
+  const rowsLength = Number(rowsLengthBig);
+  const expected = 44 + schemaLength + idsLength + (info.recordCount + 1) * 8 + rowsLength;
+  need(raw.length === expected, "RIDA catalog layout is invalid");
+  const schemaEnd = 44 + schemaLength;
+  let fields;
+  let ids;
+  try {
+    fields = JSON.parse(raw.subarray(44, schemaEnd).toString("utf8"));
+    ids = idsLength ? JSON.parse(raw.subarray(schemaEnd, schemaEnd + idsLength).toString("utf8")) : [];
+  } catch (cause) { throw new RidaError("RIDA catalog JSON is invalid", { cause }); }
+  need(Array.isArray(fields) && fields.length === fieldCount && fields.every((x) => typeof x === "string") && new Set(fields).size === fields.length, "RIDA field dictionary is invalid");
+  need(Array.isArray(ids) && (ids.length === 0 || ids.length === info.recordCount) && ids.every((x) => typeof x === "string") && new Set(ids).size === ids.length, "RIDA record-ID dictionary is invalid");
+  const offsetsStart = schemaEnd + idsLength;
+  const rowDataStart = offsetsStart + (info.recordCount + 1) * 8;
+  const rowOffsets = [];
+  for (let n = 0; n <= info.recordCount; n++) {
+    const value = raw.readBigUInt64BE(offsetsStart + n * 8);
+    need(value <= BigInt(rowsLength), "RIDA row offset is out of bounds");
+    rowOffsets.push(Number(value));
+  }
+  need(rowOffsets[0] === 0 && rowOffsets.at(-1) === rowsLength, "RIDA row offsets are incomplete");
+  for (let n = 1; n < rowOffsets.length; n++) need(rowOffsets[n] >= rowOffsets[n - 1], "RIDA row offsets are not ordered");
+  return { raw, fields, fieldIds: new Map(fields.map((name, i) => [name, i])), ids, idToRecord: ids.length ? new Map(ids.map((id, i) => [id, i])) : null, mode: modeCode === 1 ? "dense" : "sparse", recordCount: info.recordCount, rowOffsets, rowDataStart };
+}
+
+function getRecordNumber(catalog, recordId) {
+  const id = String(recordId);
+  if (catalog.idToRecord) {
+    const n = catalog.idToRecord.get(id);
+    if (n === undefined) throw new RangeError(`Unknown RIDA record ID: ${id}`);
+    return n;
+  }
+  if (!/^[1-9]\d*$/.test(id)) throw new RangeError(`Unknown RIDA record ID: ${id}`);
+  const n = Number(id) - 1;
+  if (!Number.isSafeInteger(n) || n < 0 || n >= catalog.recordCount) throw new RangeError(`Unknown RIDA record ID: ${id}`);
+  return n;
+}
+
+function locate(catalog, recordId, wantedNames) {
+  const rowNo = getRecordNumber(catalog, recordId);
+  const start = catalog.rowOffsets[rowNo];
+  const end = catalog.rowOffsets[rowNo + 1];
+  const row = catalog.raw.subarray(catalog.rowDataStart + start, catalog.rowDataStart + end);
+  let [dataOffset, pos] = readVarint(row, 0);
+  const wanted = new Map();
+  for (const name of new Set(wantedNames)) {
+    const id = catalog.fieldIds.get(name);
+    if (id === undefined) throw new RangeError(`Unknown RIDA field: ${name}`);
+    wanted.set(id, name);
+  }
+  const found = new Map();
+  if (catalog.mode === "dense") {
+    for (let fieldId = 0, valueOffset = dataOffset; fieldId < catalog.fields.length; fieldId++) {
+      let length; [length, pos] = readVarint(row, pos);
+      if (length) {
+        const name = wanted.get(fieldId);
+        if (name !== undefined) found.set(name, [valueOffset, length]);
+        valueOffset += length;
+      }
+    }
+    return found;
+  }
+  let count; [count, pos] = readVarint(row, pos);
+  let fieldId = 0;
+  for (let i = 0; i < count; i++) {
+    let delta; [delta, pos] = readVarint(row, pos);
+    let length; [length, pos] = readVarint(row, pos);
+    need(!(i > 0 && delta === 0) && length > 0, "RIDA sparse row is invalid");
+    fieldId += delta;
+    need(fieldId < catalog.fields.length, "RIDA sparse field ID is out of range");
+    const name = wanted.get(fieldId);
+    if (name !== undefined) found.set(name, [dataOffset, length]);
+    dataOffset += length;
+  }
+  return found;
+}
+
+function normalizeKey(key) {
+  if (key == null) return null;
+  if (Buffer.isBuffer(key) || key instanceof Uint8Array) {
+    const result = Buffer.from(key);
+    need(result.length === 32, "RIDA encryption key must be exactly 32 bytes");
+    return result;
+  }
+  const text = String(key).trim();
+  let result;
+  try { result = Buffer.from(text, "base64"); } catch { result = Buffer.alloc(0); }
+  if (result.length !== 32) result = Buffer.from(text, "hex");
+  need(result.length === 32, "RIDA key must be 32 bytes or its base64/hex text form");
+  return result;
+}
+
+function decodeValue(bytes, info, key) {
+  let raw = bytes;
+  if (info.encrypted) {
+    need(key, "This RIDA file is encrypted; provide its private key");
+    need(raw.length >= 28, "Encrypted RIDA value is incomplete");
+    const decipher = createDecipheriv("aes-256-gcm", key, raw.subarray(0, 12));
+    decipher.setAuthTag(raw.subarray(raw.length - 16));
+    try { raw = Buffer.concat([decipher.update(raw.subarray(12, -16)), decipher.final()]); }
+    catch (cause) { throw new RidaError("RIDA field value could not be decrypted", { cause }); }
+  }
+  if (info.valueCodec === "deflate-per-field") {
+    try { raw = inflateRawSync(raw); }
+    catch (cause) { throw new RidaError("RIDA field value cannot be decompressed", { cause }); }
+  }
+  try { return JSON.parse(raw.toString("utf8")); }
+  catch (cause) { throw new RidaError("RIDA field value is not valid JSON", { cause }); }
+}
+
+function companionPath(dataPath, suffix) {
+  const parsed = parse(dataPath);
+  const stem = extname(dataPath) ? basename(dataPath, extname(dataPath)) : parsed.base;
+  const clean = stem.endsWith("Data") ? stem.slice(0, -4) : stem;
+  return join(dirname(dataPath), `${clean}${suffix}`);
+}
+
+function remoteIndexUrl(dataUrl) {
+  const url = new URL(dataUrl);
+  const file = basename(url.pathname);
+  const extension = extname(file);
+  let stem = extension ? file.slice(0, -extension.length) : file;
+  if (stem.endsWith("Data")) stem = stem.slice(0, -4);
+  url.pathname = `${dirname(url.pathname)}/${stem}Index.rida`.replaceAll("\\", "/");
+  return url.toString();
+}
+
+async function bytesFrom(location) {
+  if (location instanceof Uint8Array) return Buffer.from(location);
+  if (location instanceof URL && location.protocol === "file:") return readFile(location);
+  if (typeof location === "string" && /^https?:\/\//i.test(location)) {
+    const response = await fetch(location);
+    if (!response.ok) throw new RidaError(`Could not fetch ${location}: HTTP ${response.status}`);
+    return Buffer.from(await response.arrayBuffer());
+  }
+  return readFile(location);
+}
+
+export class RidaReader {
+  static async open(data, { index, map, key } = {}) {
+    const dataIsUrl = typeof data === "string" && /^https?:\/\//i.test(data);
+    let indexLocation = index;
+    if (!indexLocation) {
+      if (dataIsUrl) indexLocation = remoteIndexUrl(data);
+      else indexLocation = companionPath(String(data), "Index.rida");
+    }
+    const [indexBytes, mapBytes] = await Promise.all([bytesFrom(indexLocation), map ? bytesFrom(map) : (dataIsUrl ? Promise.resolve(null) : readFile(companionPath(String(data), "Map.json")).catch(() => null))]);
+    const { info, catalog: compressed } = parseIndex(indexBytes);
+    const catalog = await parseCatalog(compressed, info);
+    const normalizedKey = normalizeKey(key);
+    need(!info.encrypted || normalizedKey, "This RIDA file is encrypted; provide its private key");
+    let mapping = null;
+    if (mapBytes) { try { mapping = JSON.parse(mapBytes.toString("utf8")); } catch { throw new RidaError("RIDA map JSON is invalid"); } }
+    let handle = null;
+    if (!dataIsUrl) {
+      handle = await open(data instanceof URL ? data : String(data), "r");
+      const stat = await handle.stat();
+      need(stat.size === info.dataLength, "RIDA data file does not match its index");
+    }
+    return new RidaReader(data, handle, info, catalog, mapping, normalizedKey);
+  }
+
+  constructor(data, handle, info, catalog, mapping, key) {
+    this.data = data; this.handle = handle; this.info = info; this.catalog = catalog; this.map = mapping; this.key = key;
+  }
+
+  async _range(offset, length) {
+    need(Number.isSafeInteger(offset) && Number.isSafeInteger(length) && offset >= 0 && length > 0 && offset + length <= this.info.dataLength, "Requested RIDA byte range is invalid");
+    if (this.handle) {
+      const result = Buffer.alloc(length);
+      const { bytesRead } = await this.handle.read(result, 0, length, offset);
+      need(bytesRead === length, "RIDA data range is incomplete");
+      return result;
+    }
+    const response = await fetch(this.data, { headers: { Range: `bytes=${offset}-${offset + length - 1}` } });
+    need(response.status === 206, `RIDA host must support HTTP byte ranges (received ${response.status})`);
+    const result = Buffer.from(await response.arrayBuffer());
+    need(result.length === length, `RIDA range returned ${result.length} bytes; expected ${length}`);
+    return result;
+  }
+
+  get fields() { return [...this.catalog.fields]; }
+  get recordCount() { return this.info.recordCount; }
+  get valueCodec() { return this.info.valueCodec; }
+  get mode() { return this.catalog.mode; }
+
+  recordsUnder(prefix) {
+    const start = `${String(prefix).replace(/\/$/, "")}/`;
+    return this.catalog.ids.filter((id) => id.startsWith(start));
+  }
+
+  resolve(...names) {
+    const wanted = names.map((name) => String(name).trim().toLocaleLowerCase()).filter(Boolean);
+    const paths = this.map?.paths;
+    if (!wanted.length || !paths || typeof paths !== "object") return null;
+    let found = null;
+    for (const [prefix, folderName] of Object.entries(paths)) {
+      const chain = [];
+      const parts = prefix.split("/");
+      for (let i = 1; i <= parts.length; i++) {
+        const key = parts.slice(0, i).join("/");
+        if (typeof paths[key] === "string") chain.push(paths[key].toLocaleLowerCase());
+      }
+      const last = wanted.at(-1);
+      const lastMatch = folderName.toLocaleLowerCase();
+      if (!(lastMatch === last || lastMatch.endsWith(last) || lastMatch.includes(last))) continue;
+      if (wanted.every((name) => chain.some((part) => part.includes(name))) && (!found || prefix.length > found.length)) found = prefix;
+    }
+    return found;
+  }
+
+  _resolveId(recordId) {
+    const text = String(recordId).replace(/\/$/, "");
+    if (!this.map || !text || this.catalog.idToRecord?.has(text)) return recordId;
+    const parts = text.split("/");
+    if (parts.length > 1 && /^\d+$/.test(parts.at(-1))) {
+      const prefix = this.resolve(...parts.slice(0, -1));
+      if (prefix) {
+        const candidate = `${prefix}/${parts.at(-1)}`;
+        if (this.catalog.idToRecord?.has(candidate)) return candidate;
+        const descendant = this.recordsUnder(prefix).find((id) => id.split("/").at(-1) === parts.at(-1));
+        if (descendant) return descendant;
+        return candidate;
+      }
+    }
+    const prefix = this.resolve(...parts);
+    if (prefix) return this.recordsUnder(prefix)[0] ?? recordId;
+    return recordId;
+  }
+
+  async get(recordId, field) {
+    recordId = this._resolveId(recordId);
+    const pointer = locate(this.catalog, recordId, [field]).get(field);
+    if (!pointer) throw new RangeError(`Field ${field} is absent from record ${recordId}`);
+    return decodeValue(await this._range(...pointer), this.info, this.key);
+  }
+
+  async getMany(recordId, fields) {
+    recordId = this._resolveId(recordId);
+    const pointers = locate(this.catalog, recordId, fields);
+    const sorted = [...pointers.entries()].sort((a, b) => a[1][0] - b[1][0]);
+    const groups = [];
+    for (const entry of sorted) {
+      const [offset, length] = entry[1];
+      const last = groups.at(-1);
+      if (last && offset - last.end <= MERGE_GAP_LIMIT) {
+        last.end = Math.max(last.end, offset + length);
+        last.entries.push(entry);
+      } else groups.push({ start: offset, end: offset + length, entries: [entry] });
+    }
+    const out = {};
+    for (const group of groups) {
+      const block = await this._range(group.start, group.end - group.start);
+      for (const [name, [offset, length]] of group.entries) {
+        out[name] = decodeValue(block.subarray(offset - group.start, offset - group.start + length), this.info, this.key);
+      }
+    }
+    return out;
+  }
+
+  async getRecord(recordId) { return this.getMany(this._resolveId(recordId), this.catalog.fields); }
+
+  async verifyData() {
+    const hash = createHash("sha256");
+    const chunkSize = 1024 * 1024;
+    for (let offset = 0; offset < this.info.dataLength; offset += chunkSize) {
+      hash.update(await this._range(offset, Math.min(chunkSize, this.info.dataLength - offset)));
+    }
+    return this.info.digest.equals(Buffer.alloc(32)) || hash.digest().equals(this.info.digest);
+  }
+
+  async close() { await this.handle?.close(); this.handle = null; }
+}
+
+export { decodeValue, parseIndex, parseCatalog };