inibase 3.6.0 → 3.7.1

package/README.md

@@ -2,22 +2,36 @@
 
 > A file-based & memory-efficient, serverless relational database with **crash-atomic ACID: single-table DML + multi-table transactions** :fire: — per-table writer locks, write-ahead journal + crash recovery, fsync-backed durability, and a `begin`/`commit`/`rollback` transaction API (exact scope in [Durability & crash safety](#durability--crash-safety)).
 
-[![Inibase banner](./.github/assets/banner.jpg)](https://github.com/inicontent/inibase)
+[![Inibase: TypeScript API, column-file storage, relations, transactions, and durability](./.github/assets/overview.svg)](#usage)
 
 [![npmjs](https://img.shields.io/npm/dm/inibase.svg?style=flat)](https://www.npmjs.org/package/inibase) [![License](https://img.shields.io/github/license/inicontent/inibase.svg?style=flat&colorA=18181B&colorB=28CF8D)](./LICENSE) [![Activity](https://img.shields.io/github/commit-activity/m/inicontent/inibase)](https://github.com/inicontent/inibase/pulse) [![GitHub stars](https://img.shields.io/github/stars/inicontent/inibase?style=social)](https://github.com/inicontent/inibase)
 
+**[Quick start](#usage) · [How it works](#how-it-works) · [Durability](#durability--crash-safety) · [Benchmarks](#benchmark)**
+
+## Small footprint. Real benchmark results.
+
+Inibase is a file-based TypeScript database designed for **fast data access and low memory usage**, with column selection, built-in caching, and optional gzip compression.
+
+In the **100,000-record CMS benchmark**, Inibase **3.7.0 with gzip** used **1.64 MiB of allocated disk** — **89.4% less than LevelDB** and **98.7% less than SQLite**. Reading two selected columns took **44.3 ms** in the same workload.
+
+[![Database storage comparison for 100,000 CMS records: Inibase gzip uses 1.64 MiB of allocated disk; lower is better](./.github/assets/storage-benchmark-100k.png)](https://github.com/inicontent/benchmarks/blob/a6c4a644aa5472192fefefe0f3d93b56edc9726a/results/large-cms/REPORT.md)
+
+*Source: [inicontent/benchmarks](https://github.com/inicontent/benchmarks/blob/a6c4a644aa5472192fefefe0f3d93b56edc9726a/results/large-cms/REPORT.md), median of three trials on Apple M5 with repetitive synthetic CMS content and full durability. The chart measures disk storage, not RAM. Results depend on the workload; indexed competitors were faster for key lookups.*
+
+[Explore the full results](https://github.com/inicontent/benchmarks/blob/a6c4a644aa5472192fefefe0f3d93b56edc9726a/results/large-cms/REPORT.md) · [Read the methodology](https://github.com/inicontent/benchmarks/blob/a6c4a644aa5472192fefefe0f3d93b56edc9726a/METHODOLOGY.md)
+
 ## Features
 
 - **Lightweight** 🪶
 - **Minimalist** :white_circle: (but powerful)
 - **100% TypeScript** :large_blue_diamond:
-- **Super-Fast** :zap: (built-in caching system)
+- **Built for fast access** :zap: Built-in caching and column projection
 - **ATOMIC** :lock: Per-table writer locks, write-ahead journal + crash recovery, fsync-backed durability, and multi-table transactions (`begin`/`commit`/`rollback`) for atomic cascades (exact scope below)
 - **Built-in** form validation (+unique values :new: ) :sunglasses:
 - **Suitable for large data** :page_with_curl: (tested with 4M records)
 - **Support Compression** :eight_spoked_asterisk: (using built-in nodejs zlib)
 - **Support Table Joins** :link:
-- **Low memory-usage** :chart_with_downwards_trend: (3-5mb)
+- **Memory-conscious design** :chart_with_downwards_trend: File-based storage and streaming reads
 - **Safe** :lock: (no sql or javascript injections)
 - **Easy to use** :bread:
 - **...** and much more :rocket:
@@ -1086,6 +1100,10 @@ are derived first, in dependency order.
 
 ### Computed fields
 
+![Bulk POST benchmark comparing plain and computed fields for 10, 100, and 1,000 rows; lower is better](./.github/assets/computed-benchmark.svg)
+
+*Charted from the results below, with fsync and journaling enabled. These are recorded sample measurements, not a new benchmark run; hardware and version were not recorded with this table.*
+
 Write-time evaluation cost (helpers `sum(3 * 4)`, `avg(4)`, `min(4)`, `max(4)`, `count(3)` over an `items` array with 3 lines, plus an arithmetic field `itemTotal × (314 / 100)`), compared against the identical table without computed fields:
 
 | rows | POST bulk (plain / computed) | POST single (plain / computed) | PUT recompute (plain / computed) |

package/dist/file.d.ts

@@ -88,6 +88,20 @@ export declare function get(filePath: string, lineNumbers: undefined | number |
     Record<number, string | number | boolean | null | undefined | (string | number | boolean | (string | number | boolean)[] | null)[]> | null,
     number
 ]>;
+/**
+ * Reads only the requested line numbers of one column file and decodes each
+ * with `field`. Unlike the shell `sed -n` reader this stops as soon as the last
+ * requested line has been seen, so a point lookup never scans the column to
+ * EOF, and it decodes each column independently (no tab-delimited `paste`
+ * alignment to break on values that contain a literal tab). Compression is
+ * handled transparently: `.gz` columns are gunzipped on the fly.
+ *
+ * Every resource (readline, gunzip/source stream, file handle) is released on
+ * both the normal and the early-exit path.
+ */
+export declare const getLines: (filePath: string, lineNumbers: number[], field?: Field & {
+    databasePath?: string;
+}) => Promise<Record<number, ReturnType<typeof decode>>>;
 /**
  * Asynchronously replaces specific lines in a file based on the provided replacements map or string.
  *
@@ -146,9 +160,16 @@ export declare const remove: (filePath: string, linesToDelete: number | number[]
  */
 export declare const search: (filePath: string, operator: ComparisonOperator | ComparisonOperator[], comparedAtValue: string | number | boolean | null | (string | number | boolean | null)[], logicalOperator?: "and" | "or", searchIn?: Set<number>, field?: Field & {
     databasePath?: string;
-}, limit?: number, offset?: number, readWholeFile?: boolean) => Promise<[Record<number, string | number | boolean | null | undefined | (string | number | boolean | null)[]> | null, number, Set<number> | null]>;
-export declare const sum: (fp: string, ln?: number | number[]) => Promise<number>;
-export declare const avg: (fp: string, ln?: number | number[]) => Promise<number>;
-export declare const min: (fp: string, ln?: number | number[]) => Promise<number>;
-export declare const max: (fp: string, ln?: number | number[]) => Promise<number>;
+}, limit?: number, offset?: number, readWholeFile?: boolean, earlyStop?: boolean, 
+/**
+ * Per-line raw cell overrides (update-log overlay). When a line is present
+ * the overridden raw value is decoded and compared instead of the file's
+ * line, so a search reflects un-compacted `put`s. The native whole-line
+ * grep fast path is bypassed whenever overrides are supplied.
+ */
+overrides?: Map<number, string | number | boolean | null>) => Promise<[Record<number, string | number | boolean | null | undefined | (string | number | boolean | null)[]> | null, number, Set<number> | null]>;
+export declare const sum: (fp: string, ln?: number | number[], overrides?: Map<number, string | number | boolean | null>) => Promise<number>;
+export declare const avg: (fp: string, ln?: number | number[], overrides?: Map<number, string | number | boolean | null>) => Promise<number>;
+export declare const min: (fp: string, ln?: number | number[], overrides?: Map<number, string | number | boolean | null>) => Promise<number>;
+export declare const max: (fp: string, ln?: number | number[], overrides?: Map<number, string | number | boolean | null>) => Promise<number>;
 export declare const getFileDate: (path: string) => Promise<Date>;

package/dist/file.js

@@ -343,6 +343,16 @@ export function escapeShellPath(filePath) {
  */
 let tempSeq = 0;
 const tempPathFor = (filePath) => filePath.replace(/([^/]+)\/?$/, `.tmp/$1.${process.pid}-${(tempSeq++).toString(36)}`);
+/**
+ * Cleanup-and-rethrow for a failed column write. A partial temp file must never
+ * be left behind, and the caller must observe the failure so it can abort the
+ * operation instead of publishing a column that is missing a line — which would
+ * desync the pagination row count from the column files.
+ */
+const failWrite = async (fileTempPath, error) => {
+    await unlink(fileTempPath).catch(() => { });
+    throw error;
+};
 const _pipeline = async (filePath, rl, writeStream, transform) => {
     if (filePath.endsWith(".gz"))
         await pipeline(rl, transform, createGzip(), writeStream);
@@ -361,6 +371,30 @@ const createReadLineInternface = (filePath, fileHandle) => createInterface({
         : fileHandle.createReadStream(),
     crlfDelay: Number.POSITIVE_INFINITY,
 });
+/**
+ * Like {@link createReadLineInternface} but also hands back a disposer that
+ * tears down the readline interface and both the source and (for `.gz`
+ * columns) gunzip streams. Sparse readers call the disposer in their `finally`
+ * so an early `break` — the whole point of a point lookup — cannot leave a read
+ * stream (and its libuv handle) dangling, which would keep the process alive.
+ */
+const openReadline = (filePath, fileHandle) => {
+    const source = fileHandle.createReadStream();
+    const input = filePath.endsWith(".gz") ? source.pipe(createGunzip()) : source;
+    const rl = createInterface({
+        input,
+        crlfDelay: Number.POSITIVE_INFINITY,
+    });
+    return {
+        rl,
+        destroy: () => {
+            rl.close();
+            input.destroy();
+            if (input !== source)
+                source.destroy();
+        },
+    };
+};
 /**
  * Checks if a file or directory exists at the specified path.
  *
@@ -619,6 +653,45 @@ export async function get(filePath, lineNumbers, field, readWholeFile = false) {
         await fileHandle?.close();
     }
 }
+/**
+ * Reads only the requested line numbers of one column file and decodes each
+ * with `field`. Unlike the shell `sed -n` reader this stops as soon as the last
+ * requested line has been seen, so a point lookup never scans the column to
+ * EOF, and it decodes each column independently (no tab-delimited `paste`
+ * alignment to break on values that contain a literal tab). Compression is
+ * handled transparently: `.gz` columns are gunzipped on the fly.
+ *
+ * Every resource (readline, gunzip/source stream, file handle) is released on
+ * both the normal and the early-exit path.
+ */
+export const getLines = async (filePath, lineNumbers, field) => {
+    const lines = {};
+    if (!lineNumbers.length)
+        return lines;
+    const wanted = new Set(lineNumbers);
+    const last = Math.max(...lineNumbers);
+    const config = field ?? {
+        key: "BLABLA",
+        type: "string",
+    };
+    const fileHandle = await open(filePath, "r");
+    const { rl, destroy } = openReadline(filePath, fileHandle);
+    try {
+        let linesCount = 0;
+        for await (const line of rl) {
+            linesCount++;
+            if (wanted.has(linesCount))
+                lines[linesCount] = decode(line, config);
+            if (linesCount >= last)
+                break;
+        }
+        return lines;
+    }
+    finally {
+        destroy();
+        await fileHandle.close().catch(() => { });
+    }
+};
 /**
  * Asynchronously replaces specific lines in a file based on the provided replacements map or string.
  *
@@ -675,8 +748,8 @@ export const replace = async (filePath, replacements, totalItems) => {
                 }));
                 return [fileTempPath, filePath];
             }
-            catch {
-                return [fileTempPath, null];
+            catch (error) {
+                return failWrite(fileTempPath, error);
             }
             finally {
                 // Ensure that file handles are closed, even if an error occurred
@@ -695,8 +768,8 @@ export const replace = async (filePath, replacements, totalItems) => {
                 await exec(command);
                 return [fileTempPath, filePath];
             }
-            catch {
-                return [fileTempPath, null];
+            catch (error) {
+                return failWrite(fileTempPath, error);
             }
         }
     }
@@ -736,8 +809,8 @@ export const replace = async (filePath, replacements, totalItems) => {
             }
             return [fileTempPath, filePath];
         }
-        catch {
-            return [fileTempPath, null];
+        catch (error) {
+            return failWrite(fileTempPath, error);
         }
     }
     return [];
@@ -759,10 +832,12 @@ export const append = async (filePath, data) => {
                 await appendFile(fileTempPath, `${Array.isArray(data) ? data.join("\n") : data}\n`);
             }
             else {
-                const escapedFileTempPath = escapeShellPath(fileTempPath);
-                await exec(`echo '${(Array.isArray(data) ? data.join("\n") : data)
-                    .toString()
-                    .replace(/'/g, "\\'")}' | gzip - >> ${escapedFileTempPath}`);
+                // Append a complete gzip member. Readers gunzip every member in
+                // sequence (multi-member stream), and this avoids the shell's
+                // argument-length limits and single-quote escaping the old
+                // `echo … | gzip - >> file` pipeline suffered from.
+                const text = `${Array.isArray(data) ? data.join("\n") : data}\n`;
+                await appendFile(fileTempPath, await gzip(text));
             }
         }
         else
@@ -775,8 +850,8 @@ export const append = async (filePath, data) => {
                 : `${Array.isArray(data) ? data.join("\n") : data}\n`);
         return [fileTempPath, filePath];
     }
-    catch {
-        return [fileTempPath, null];
+    catch (error) {
+        return failWrite(fileTempPath, error);
     }
 };
 /**
@@ -808,8 +883,8 @@ export const prepend = async (filePath, data) => {
                     },
                 }));
             }
-            catch {
-                return [fileTempPath, null];
+            catch (error) {
+                return failWrite(fileTempPath, error);
             }
             finally {
                 // Ensure that file handles are closed, even if an error occurred
@@ -826,8 +901,8 @@ export const prepend = async (filePath, data) => {
                 const escapedFileChildTempPath = escapeShellPath(fileChildTempPath);
                 await exec(`cat ${escapedFileChildTempPath} ${escapedFilePath} > ${escapedFileTempPath}`);
             }
-            catch {
-                return [fileTempPath, null];
+            catch (error) {
+                return failWrite(fileTempPath, error);
             }
             finally {
                 await unlink(fileChildTempPath);
@@ -843,8 +918,8 @@ export const prepend = async (filePath, data) => {
                 ? await gzip(`${Array.isArray(data) ? data.join("\n") : data}\n`)
                 : `${Array.isArray(data) ? data.join("\n") : data}\n`);
         }
-        catch {
-            return [fileTempPath, null];
+        catch (error) {
+            return failWrite(fileTempPath, error);
         }
     }
     return [fileTempPath, filePath];
@@ -874,8 +949,8 @@ export const remove = async (filePath, linesToDelete) => {
         await exec(command);
         return [fileTempPath, filePath];
     }
-    catch {
-        return [fileTempPath, null];
+    catch (error) {
+        return failWrite(fileTempPath, error);
     }
 };
 /**
@@ -1094,9 +1169,17 @@ async function searchEqualsNative(filePath, patterns, field, searchIn, comparedA
  *
  * Note: Decodes each line for comparison and can handle complex queries with multiple conditions.
  */
-export const search = async (filePath, operator, comparedAtValue, logicalOperator, searchIn, field, limit, offset, readWholeFile) => {
+export const search = async (filePath, operator, comparedAtValue, logicalOperator, searchIn, field, limit, offset, readWholeFile, earlyStop, 
+/**
+ * Per-line raw cell overrides (update-log overlay). When a line is present
+ * the overridden raw value is decoded and compared instead of the file's
+ * line, so a search reflects un-compacted `put`s. The native whole-line
+ * grep fast path is bypassed whenever overrides are supplied.
+ */
+overrides) => {
     // Native fast path for exact-equality searches (whole-line `grep -x -F`).
-    if (operator === "=" &&
+    if (!overrides?.size &&
+        operator === "=" &&
         !Array.isArray(operator) &&
         !logicalOperator &&
         comparedAtValue !== null &&
@@ -1116,6 +1199,7 @@ export const search = async (filePath, operator, comparedAtValue, logicalOperato
     let linesCount = 0;
     const linesNumbers = new Set();
     let fileHandle = null;
+    let destroyStreams = () => { };
     const config = field ?? {
         key: "BLABLA",
         type: "string",
@@ -1131,7 +1215,8 @@ export const search = async (filePath, operator, comparedAtValue, logicalOperato
         // Open the file for reading.
         fileHandle = await open(filePath, "r");
         // Create a Readline interface to read the file line by line.
-        const rl = createReadLineInternface(filePath, fileHandle);
+        const { rl, destroy } = openReadline(filePath, fileHandle);
+        destroyStreams = destroy;
         // Iterate through each line in the file.
         for await (const line of rl) {
             // Increment the line count for each line.
@@ -1140,8 +1225,11 @@ export const search = async (filePath, operator, comparedAtValue, logicalOperato
             if (searchIn?.size &&
                 (!searchIn.has(linesCount) || searchIn.has(-linesCount)))
                 continue;
-            // Decode the line for comparison.
-            const decodedLine = decode(line, config);
+            // Decode the line for comparison. A per-line overlay override (raw
+            // cell) wins over the file content so searches see un-compacted puts.
+            const decodedLine = decode(overrides?.has(linesCount)
+                ? overrides.get(linesCount)
+                : line, config);
             // Check if the line meets the specified conditions based on comparison and logical operators.
             const doesMeetCondition = (Array.isArray(decodedLine) &&
                 operator !== "=" &&
@@ -1162,6 +1250,10 @@ export const search = async (filePath, operator, comparedAtValue, logicalOperato
                 }
                 // Store the decoded line in the result object.
                 matchingLines[linesCount] = decodedLine;
+                // Every requested match is now in hand: stop instead of
+                // scanning the rest of the column (id -> line resolution).
+                if (earlyStop && limit && linesNumbers.size >= limit)
+                    break;
             }
         }
         // Convert the Map to an object using Object.fromEntries and return the result.
@@ -1173,7 +1265,9 @@ export const search = async (filePath, operator, comparedAtValue, logicalOperato
         return [null, 0, null];
     }
     finally {
-        // Close the file handle in the finally block to ensure it is closed even if an error occurs.
+        // Tear down the read streams (an early `break` must not leave a handle
+        // open) and then the file handle, even if an error occurred.
+        destroyStreams();
         await fileHandle?.close();
     }
 };
@@ -1187,7 +1281,7 @@ export const search = async (filePath, operator, comparedAtValue, logicalOperato
  *
  * @returns Promise<number>  The requested metric, or 0 if no numeric value found.
  */
-async function reduceNumbers(filePath, wanted = "sum", lineNumbers) {
+async function reduceNumbers(filePath, wanted = "sum", lineNumbers, overrides) {
     /* optional subset */
     const filter = lineNumbers
         ? new Set(Array.isArray(lineNumbers) ? lineNumbers : [lineNumbers])
@@ -1207,7 +1301,13 @@ async function reduceNumbers(filePath, wanted = "sum", lineNumbers) {
             /* skip unwanted lines */
             if (filter && !filter.has(line))
                 continue;
-            const num = Number(decode(txt, { key: "BLABLA", type: "number" }));
+            // A per-line overlay override (raw cell) wins over the base file so
+            // aggregates reflect un-compacted puts.
+            const raw = overrides?.has(line) ? overrides.get(line) : txt;
+            const num = Number(decode(raw, {
+                key: "BLABLA",
+                type: "number",
+            }));
             if (Number.isNaN(num))
                 continue;
             processed++;
@@ -1241,10 +1341,10 @@ async function reduceNumbers(filePath, wanted = "sum", lineNumbers) {
                 : max;
 }
 /* Optional convenience wrappers (signatures unchanged) */
-export const sum = (fp, ln) => reduceNumbers(fp, "sum", ln);
-export const avg = (fp, ln) => reduceNumbers(fp, "avg", ln);
-export const min = (fp, ln) => reduceNumbers(fp, "min", ln);
-export const max = (fp, ln) => reduceNumbers(fp, "max", ln);
+export const sum = (fp, ln, overrides) => reduceNumbers(fp, "sum", ln, overrides);
+export const avg = (fp, ln, overrides) => reduceNumbers(fp, "avg", ln, overrides);
+export const min = (fp, ln, overrides) => reduceNumbers(fp, "min", ln, overrides);
+export const max = (fp, ln, overrides) => reduceNumbers(fp, "max", ln, overrides);
 export const getFileDate = (path) => stat(path)
     .then((s) => s.mtime || s.birthtime)
     .catch(() => new Date());

package/dist/format.d.ts

@@ -0,0 +1,54 @@
+/**
+ * On-disk format marker for a table directory.
+ *
+ * Format 1 is the implicit legacy format (no `.format` file): base columns plus
+ * pagination only. Format 2 opts the table into the append-only update log
+ * written by `put` (see `updatelog.ts`).
+ *
+ * The marker is the single source of truth for "does this table have a log?".
+ * It also carries the compatibility contract read by a later 3.7.1 detector:
+ * `minReader` is the oldest engine that may open the table, `detector` the
+ * release that hard-fails instead of silently mis-reading. `baseGen` is the
+ * generation the base columns were last compacted at; it is bumped by
+ * compaction (3.9) so a stale in-memory overlay can detect that its base moved
+ * and rebuild instead of replaying frames that are already merged.
+ */
+export declare const FORMAT_FILE = ".format";
+export interface TableFormat {
+    format: number;
+    log: boolean;
+    baseGen: number;
+    /** Oldest engine that may open the table, as written into the marker. */
+    minReader?: string;
+}
+export declare const LEGACY_FORMAT: TableFormat;
+export declare const CURRENT_FORMAT_VERSION = 2;
+export declare const ENGINE_NAME = "inibase";
+/** This build's engine version, compared against a marker's `minReader`. */
+export declare const ENGINE_VERSION = "3.8.0";
+/** Oldest reader that understands format 2. */
+export declare const MIN_READER = "3.8.0";
+/** Release expected to detect (and refuse) format 2 rather than mis-read it. */
+export declare const DETECTOR = "3.7.1";
+/** Compare dotted numeric versions: >0 when `a > b`, <0 when `a < b`, else 0. */
+export declare const compareVersions: (a: string, b: string) => number;
+/**
+ * Why this engine cannot open a marker, or null when it can.
+ *
+ * Refusing *before* any column is read keeps a newer-format table from being
+ * silently mis-read as format 2 (or legacy). Two independent conditions:
+ * a `format` above what this build writes, or a `minReader` above the running
+ * engine version.
+ */
+export declare const unsupportedFormatReason: (format: TableFormat) => string | null;
+/**
+ * Read a table's format marker. A missing or unreadable marker is legacy
+ * (format 1), which keeps every pre-3.8 table byte-for-byte compatible.
+ */
+export declare const readFormat: (tablePath: string) => Promise<TableFormat>;
+/** The exact bytes of a format marker (shared by write and compaction commit). */
+export declare const serializeFormat: (format: TableFormat) => string;
+/** Write (or replace) the format marker, fsync-backed. */
+export declare const writeFormat: (tablePath: string, format: TableFormat) => Promise<void>;
+/** Remove the marker (used when downgrading a table back to legacy). */
+export declare const removeFormat: (tablePath: string) => Promise<void>;

package/dist/format.js

@@ -0,0 +1,108 @@
+import { readFile, unlink } from "node:fs/promises";
+import { join } from "node:path";
+import * as File from "./file.js";
+/**
+ * On-disk format marker for a table directory.
+ *
+ * Format 1 is the implicit legacy format (no `.format` file): base columns plus
+ * pagination only. Format 2 opts the table into the append-only update log
+ * written by `put` (see `updatelog.ts`).
+ *
+ * The marker is the single source of truth for "does this table have a log?".
+ * It also carries the compatibility contract read by a later 3.7.1 detector:
+ * `minReader` is the oldest engine that may open the table, `detector` the
+ * release that hard-fails instead of silently mis-reading. `baseGen` is the
+ * generation the base columns were last compacted at; it is bumped by
+ * compaction (3.9) so a stale in-memory overlay can detect that its base moved
+ * and rebuild instead of replaying frames that are already merged.
+ */
+export const FORMAT_FILE = ".format";
+export const LEGACY_FORMAT = {
+    format: 1,
+    log: false,
+    baseGen: 0,
+};
+export const CURRENT_FORMAT_VERSION = 2;
+export const ENGINE_NAME = "inibase";
+/** This build's engine version, compared against a marker's `minReader`. */
+export const ENGINE_VERSION = "3.8.0";
+/** Oldest reader that understands format 2. */
+export const MIN_READER = "3.8.0";
+/** Release expected to detect (and refuse) format 2 rather than mis-read it. */
+export const DETECTOR = "3.7.1";
+/** Compare dotted numeric versions: >0 when `a > b`, <0 when `a < b`, else 0. */
+export const compareVersions = (a, b) => {
+    const pa = a.split(".");
+    const pb = b.split(".");
+    for (let i = 0; i < Math.max(pa.length, pb.length); i++) {
+        const diff = (Number(pa[i]) || 0) - (Number(pb[i]) || 0);
+        if (diff !== 0)
+            return diff > 0 ? 1 : -1;
+    }
+    return 0;
+};
+/**
+ * Why this engine cannot open a marker, or null when it can.
+ *
+ * Refusing *before* any column is read keeps a newer-format table from being
+ * silently mis-read as format 2 (or legacy). Two independent conditions:
+ * a `format` above what this build writes, or a `minReader` above the running
+ * engine version.
+ */
+export const unsupportedFormatReason = (format) => {
+    if (format.format > CURRENT_FORMAT_VERSION)
+        return `format ${format.format} (this engine reads up to ${CURRENT_FORMAT_VERSION})`;
+    if (format.minReader && compareVersions(format.minReader, ENGINE_VERSION) > 0)
+        return `minReader ${format.minReader} (this engine is ${ENGINE_VERSION})`;
+    return null;
+};
+const formatPath = (tablePath) => join(tablePath, FORMAT_FILE);
+/**
+ * Read a table's format marker. A missing or unreadable marker is legacy
+ * (format 1), which keeps every pre-3.8 table byte-for-byte compatible.
+ */
+export const readFormat = async (tablePath) => {
+    if (!(await File.isExists(formatPath(tablePath))))
+        return { ...LEGACY_FORMAT };
+    try {
+        const raw = JSON.parse(await readFile(formatPath(tablePath), "utf8"));
+        return {
+            // Accept both the current `format` key and the earlier `version`
+            // spelling so a marker written by any 3.8 prerelease still opens.
+            format: Number(raw?.format ?? raw?.version) || LEGACY_FORMAT.format,
+            log: raw?.log === true,
+            baseGen: Number(raw?.baseGen) || 0,
+            minReader: typeof raw?.minReader === "string" ? raw.minReader : undefined,
+        };
+    }
+    catch {
+        // A corrupt marker means we cannot trust "no log": refusing to write
+        // over it is safer, but reading must still work, so treat as legacy and
+        // let the write path surface UNSUPPORTED_FORMAT.
+        return { ...LEGACY_FORMAT };
+    }
+};
+/** The exact bytes of a format marker (shared by write and compaction commit). */
+export const serializeFormat = (format) => `${JSON.stringify({
+    engine: ENGINE_NAME,
+    format: format.format,
+    minReader: MIN_READER,
+    detector: DETECTOR,
+    layout: "append",
+    log: format.log,
+    baseGen: format.baseGen,
+})}\n`;
+/** Write (or replace) the format marker, fsync-backed. */
+export const writeFormat = async (tablePath, format) => {
+    await File.write(formatPath(tablePath), serializeFormat(format));
+};
+/** Remove the marker (used when downgrading a table back to legacy). */
+export const removeFormat = async (tablePath) => {
+    const path = formatPath(tablePath);
+    try {
+        await unlink(path);
+    }
+    catch {
+        // Already absent.
+    }
+};