inibase 3.7.0 → 3.8.0

package/README.md

@@ -2,22 +2,37 @@
 
 > A file-based & memory-efficient, serverless relational database with **crash-atomic ACID: single-table DML + multi-table transactions** :fire: — per-table writer locks, write-ahead journal + crash recovery, fsync-backed durability, and a `begin`/`commit`/`rollback` transaction API (exact scope in [Durability & crash safety](#durability--crash-safety)).
 
-[![Inibase banner](./.github/assets/banner.jpg)](https://github.com/inicontent/inibase)
+[![Inibase: TypeScript API, column-file storage, relations, transactions, and durability](./.github/assets/overview.svg)](#usage)
 
 [![npmjs](https://img.shields.io/npm/dm/inibase.svg?style=flat)](https://www.npmjs.org/package/inibase) [![License](https://img.shields.io/github/license/inicontent/inibase.svg?style=flat&colorA=18181B&colorB=28CF8D)](./LICENSE) [![Activity](https://img.shields.io/github/commit-activity/m/inicontent/inibase)](https://github.com/inicontent/inibase/pulse) [![GitHub stars](https://img.shields.io/github/stars/inicontent/inibase?style=social)](https://github.com/inicontent/inibase)
 
+**[Quick start](#usage) · [How it works](#how-it-works) · [Durability](#durability--crash-safety) · [Benchmarks](#benchmark)**
+
+## Small footprint. Real benchmark results.
+
+Inibase is a file-based TypeScript database designed for **fast data access and low memory usage**, with column selection, built-in caching, and optional gzip compression.
+
+In the **100,000-record CMS benchmark**, Inibase **3.7.0 with gzip** used **1.64 MiB of allocated disk** — **89.4% less than LevelDB** and **98.7% less than SQLite**. Reading two selected columns took **44.3 ms** in the same workload.
+
+[![Database storage comparison for 100,000 CMS records: Inibase gzip uses 1.64 MiB of allocated disk; lower is better](./.github/assets/storage-benchmark-100k.png)](https://github.com/inicontent/benchmarks/blob/a6c4a644aa5472192fefefe0f3d93b56edc9726a/results/large-cms/REPORT.md)
+
+*Source: [inicontent/benchmarks](https://github.com/inicontent/benchmarks/blob/a6c4a644aa5472192fefefe0f3d93b56edc9726a/results/large-cms/REPORT.md), median of three trials on Apple M5 with repetitive synthetic CMS content and full durability. The chart measures disk storage, not RAM. Results depend on the workload; indexed competitors were faster for key lookups.*
+
+[Explore the full results](https://github.com/inicontent/benchmarks/blob/a6c4a644aa5472192fefefe0f3d93b56edc9726a/results/large-cms/REPORT.md) · [Read the methodology](https://github.com/inicontent/benchmarks/blob/a6c4a644aa5472192fefefe0f3d93b56edc9726a/METHODOLOGY.md)
+
 ## Features
 
 - **Lightweight** 🪶
 - **Minimalist** :white_circle: (but powerful)
 - **100% TypeScript** :large_blue_diamond:
-- **Super-Fast** :zap: (built-in caching system)
+- **Built for fast access** :zap: Built-in caching and column projection
 - **ATOMIC** :lock: Per-table writer locks, write-ahead journal + crash recovery, fsync-backed durability, and multi-table transactions (`begin`/`commit`/`rollback`) for atomic cascades (exact scope below)
 - **Built-in** form validation (+unique values :new: ) :sunglasses:
 - **Suitable for large data** :page_with_curl: (tested with 4M records)
+- **Incremental updates** :recycle: Opt-in update log (`log: true`) appends small frames instead of rewriting whole columns
 - **Support Compression** :eight_spoked_asterisk: (using built-in nodejs zlib)
 - **Support Table Joins** :link:
-- **Low memory-usage** :chart_with_downwards_trend: (3-5mb)
+- **Memory-conscious design** :chart_with_downwards_trend: File-based storage and streaming reads
 - **Safe** :lock: (no sql or javascript injections)
 - **Easy to use** :bread:
 - **...** and much more :rocket:
@@ -70,7 +85,7 @@ const users = await db.get("user", { favoriteFoods: "![]Pizza,Burger" });
 
 - **POST**: New data is appended to column files efficiently.
 - **GET**: Data retrieval is optimized by reading files line-by-line.
-- **PUT**: Updates are streamlined, with only the relevant file being modified.
+- **PUT**: Updates are streamlined: on a plain table only the relevant column file is rewritten, while a table with the [update log](#incremental-update-log) enabled appends a small frame instead.
 - **DELETE**: Removes lines from column files for swift deletion.
 
 This structure ensures efficient storage, retrieval, and updates, making our system scalable and high-performing for diverse datasets and applications.
@@ -167,6 +182,50 @@ live-reader and stale-lock test suite, `pnpm test:transaction` for the
 transaction/cascade/crash-recovery suite, and `pnpm benchmark:durability` for
 the `full` vs `none` throughput comparison.
 
+## Incremental update log
+
+Opt a table into the append-only **update log** (on-disk format 2) to make `put`
+cheap when a table is large but each update touches few rows:
+
+```js
+await db.createTable("user", userTableSchema, { log: true });
+// or migrate an existing table in place:
+await db.updateTable("user", undefined, { log: true });
+```
+
+- **Writes.** `put` appends one length-prefixed, `crc32`-checked frame
+  (`{ id, cols, updatedAt }`) to `updates.log` instead of rewriting column
+  files, so the fast path is proportional to the number of updated rows rather
+  than the table size. `post` and `delete` still rewrite the base columns.
+- **Reads.** Every read — `get`, criteria, `columns`, sort, pagination,
+  aggregates, and uniqueness checks — merges the log over the base columns, so
+  results match an un-logged table. With an empty log the merge is a no-op.
+- **Compaction.** `await db.compact(tableName)` folds the log into the base
+  columns and resets it, returning the new `baseGen` — or `null` when there was
+  nothing to fold. Compaction is crash-safe and semantically neutral: it only
+  materializes already-acknowledged updates. It also runs automatically under
+  backpressure so the overlay and log stay bounded
+  (`INIBASE_MAX_OVERLAY_RECORDS`, default `100000`; `INIBASE_MAX_LOG_BYTES`,
+  default 64 MiB); if the bound is hit and compaction cannot run, the write
+  fails with `MAINTENANCE_REQUIRED` rather than growing without limit.
+- **Requirements.** The table must use the append layout (no `prepend`) and a
+  schema of simple top-level fields. Transactions, schema changes, and
+  disabling the log on a log table throw `MAINTENANCE_REQUIRED` for now (they
+  need compaction rewiring, planned for 3.9).
+- **On disk.** A `.format` marker records `format`, `baseGen`, and the
+  compatibility contract (`minReader`); `updates.log` holds the frames. A table
+  with no `.format` is legacy format 1 and behaves exactly as before.
+- **Downgrade safety.** A reader older than the log format ignores the marker
+  and would silently serve **stale** base columns. The marker's `minReader`
+  makes a conforming engine refuse such a table with `UNSUPPORTED_FORMAT`
+  instead of mis-reading it.
+
+> [!WARNING]
+> **Backups.** A raw directory copy is only a valid backup when writes are
+> quiesced or the copy is a filesystem snapshot. Copying a logged table *during*
+> writes can capture base columns paired with a mismatched `updates.log` /
+> `baseGen`. Running `compact` first does not remove the need to quiesce writes.
+
 ## Inibase CLI
 
 ```shell
@@ -1086,6 +1145,10 @@ are derived first, in dependency order.
 
 ### Computed fields
 
+![Bulk POST benchmark comparing plain and computed fields for 10, 100, and 1,000 rows; lower is better](./.github/assets/computed-benchmark.svg)
+
+*Charted from the results below, with fsync and journaling enabled. These are recorded sample measurements, not a new benchmark run; hardware and version were not recorded with this table.*
+
 Write-time evaluation cost (helpers `sum(3 * 4)`, `avg(4)`, `min(4)`, `max(4)`, `count(3)` over an `items` array with 3 lines, plus an arithmetic field `itemTotal × (314 / 100)`), compared against the identical table without computed fields:
 
 | rows | POST bulk (plain / computed) | POST single (plain / computed) | PUT recompute (plain / computed) |

package/dist/file.d.ts

@@ -160,9 +160,16 @@ export declare const remove: (filePath: string, linesToDelete: number | number[]
  */
 export declare const search: (filePath: string, operator: ComparisonOperator | ComparisonOperator[], comparedAtValue: string | number | boolean | null | (string | number | boolean | null)[], logicalOperator?: "and" | "or", searchIn?: Set<number>, field?: Field & {
     databasePath?: string;
-}, limit?: number, offset?: number, readWholeFile?: boolean, earlyStop?: boolean) => Promise<[Record<number, string | number | boolean | null | undefined | (string | number | boolean | null)[]> | null, number, Set<number> | null]>;
-export declare const sum: (fp: string, ln?: number | number[]) => Promise<number>;
-export declare const avg: (fp: string, ln?: number | number[]) => Promise<number>;
-export declare const min: (fp: string, ln?: number | number[]) => Promise<number>;
-export declare const max: (fp: string, ln?: number | number[]) => Promise<number>;
+}, limit?: number, offset?: number, readWholeFile?: boolean, earlyStop?: boolean, 
+/**
+ * Per-line raw cell overrides (update-log overlay). When a line is present
+ * the overridden raw value is decoded and compared instead of the file's
+ * line, so a search reflects un-compacted `put`s. The native whole-line
+ * grep fast path is bypassed whenever overrides are supplied.
+ */
+overrides?: Map<number, string | number | boolean | null>) => Promise<[Record<number, string | number | boolean | null | undefined | (string | number | boolean | null)[]> | null, number, Set<number> | null]>;
+export declare const sum: (fp: string, ln?: number | number[], overrides?: Map<number, string | number | boolean | null>) => Promise<number>;
+export declare const avg: (fp: string, ln?: number | number[], overrides?: Map<number, string | number | boolean | null>) => Promise<number>;
+export declare const min: (fp: string, ln?: number | number[], overrides?: Map<number, string | number | boolean | null>) => Promise<number>;
+export declare const max: (fp: string, ln?: number | number[], overrides?: Map<number, string | number | boolean | null>) => Promise<number>;
 export declare const getFileDate: (path: string) => Promise<Date>;

package/dist/file.js

@@ -1169,9 +1169,17 @@ async function searchEqualsNative(filePath, patterns, field, searchIn, comparedA
  *
  * Note: Decodes each line for comparison and can handle complex queries with multiple conditions.
  */
-export const search = async (filePath, operator, comparedAtValue, logicalOperator, searchIn, field, limit, offset, readWholeFile, earlyStop) => {
+export const search = async (filePath, operator, comparedAtValue, logicalOperator, searchIn, field, limit, offset, readWholeFile, earlyStop, 
+/**
+ * Per-line raw cell overrides (update-log overlay). When a line is present
+ * the overridden raw value is decoded and compared instead of the file's
+ * line, so a search reflects un-compacted `put`s. The native whole-line
+ * grep fast path is bypassed whenever overrides are supplied.
+ */
+overrides) => {
     // Native fast path for exact-equality searches (whole-line `grep -x -F`).
-    if (operator === "=" &&
+    if (!overrides?.size &&
+        operator === "=" &&
         !Array.isArray(operator) &&
         !logicalOperator &&
         comparedAtValue !== null &&
@@ -1217,8 +1225,11 @@ export const search = async (filePath, operator, comparedAtValue, logicalOperato
             if (searchIn?.size &&
                 (!searchIn.has(linesCount) || searchIn.has(-linesCount)))
                 continue;
-            // Decode the line for comparison.
-            const decodedLine = decode(line, config);
+            // Decode the line for comparison. A per-line overlay override (raw
+            // cell) wins over the file content so searches see un-compacted puts.
+            const decodedLine = decode(overrides?.has(linesCount)
+                ? overrides.get(linesCount)
+                : line, config);
             // Check if the line meets the specified conditions based on comparison and logical operators.
             const doesMeetCondition = (Array.isArray(decodedLine) &&
                 operator !== "=" &&
@@ -1270,7 +1281,7 @@ export const search = async (filePath, operator, comparedAtValue, logicalOperato
  *
  * @returns Promise<number>  The requested metric, or 0 if no numeric value found.
  */
-async function reduceNumbers(filePath, wanted = "sum", lineNumbers) {
+async function reduceNumbers(filePath, wanted = "sum", lineNumbers, overrides) {
     /* optional subset */
     const filter = lineNumbers
         ? new Set(Array.isArray(lineNumbers) ? lineNumbers : [lineNumbers])
@@ -1290,7 +1301,13 @@ async function reduceNumbers(filePath, wanted = "sum", lineNumbers) {
             /* skip unwanted lines */
             if (filter && !filter.has(line))
                 continue;
-            const num = Number(decode(txt, { key: "BLABLA", type: "number" }));
+            // A per-line overlay override (raw cell) wins over the base file so
+            // aggregates reflect un-compacted puts.
+            const raw = overrides?.has(line) ? overrides.get(line) : txt;
+            const num = Number(decode(raw, {
+                key: "BLABLA",
+                type: "number",
+            }));
             if (Number.isNaN(num))
                 continue;
             processed++;
@@ -1324,10 +1341,10 @@ async function reduceNumbers(filePath, wanted = "sum", lineNumbers) {
                 : max;
 }
 /* Optional convenience wrappers (signatures unchanged) */
-export const sum = (fp, ln) => reduceNumbers(fp, "sum", ln);
-export const avg = (fp, ln) => reduceNumbers(fp, "avg", ln);
-export const min = (fp, ln) => reduceNumbers(fp, "min", ln);
-export const max = (fp, ln) => reduceNumbers(fp, "max", ln);
+export const sum = (fp, ln, overrides) => reduceNumbers(fp, "sum", ln, overrides);
+export const avg = (fp, ln, overrides) => reduceNumbers(fp, "avg", ln, overrides);
+export const min = (fp, ln, overrides) => reduceNumbers(fp, "min", ln, overrides);
+export const max = (fp, ln, overrides) => reduceNumbers(fp, "max", ln, overrides);
 export const getFileDate = (path) => stat(path)
     .then((s) => s.mtime || s.birthtime)
     .catch(() => new Date());

package/dist/format.d.ts

@@ -0,0 +1,60 @@
+/**
+ * On-disk format marker for a table directory.
+ *
+ * Format 1 is the implicit legacy format (no `.format` file): base columns plus
+ * pagination only. Format 2 opts the table into the append-only update log
+ * written by `put` (see `updatelog.ts`).
+ *
+ * The marker is the single source of truth for "does this table have a log?".
+ * It also carries the compatibility contract read by a later 3.7.1 detector:
+ * `minReader` is the oldest engine that may open the table, `detector` the
+ * release that hard-fails instead of silently mis-reading. `baseGen` is the
+ * generation the base columns were last compacted at; it is bumped by
+ * compaction (3.9) so a stale in-memory overlay can detect that its base moved
+ * and rebuild instead of replaying frames that are already merged.
+ */
+export declare const FORMAT_FILE = ".format";
+export interface TableFormat {
+    format: number;
+    log: boolean;
+    baseGen: number;
+    /** Oldest engine that may open the table, as written into the marker. */
+    minReader?: string;
+}
+export declare const LEGACY_FORMAT: TableFormat;
+export declare const CURRENT_FORMAT_VERSION = 2;
+export declare const ENGINE_NAME = "inibase";
+/** This build's engine version, compared against a marker's `minReader`. */
+export declare const ENGINE_VERSION = "3.8.0";
+/** Oldest reader that understands format 2. */
+export declare const MIN_READER = "3.8.0";
+/** Release expected to detect (and refuse) format 2 rather than mis-read it. */
+export declare const DETECTOR = "3.7.1";
+/** Compare dotted numeric versions: >0 when `a > b`, <0 when `a < b`, else 0. */
+export declare const compareVersions: (a: string, b: string) => number;
+/**
+ * Why this engine cannot open a marker, or null when it can.
+ *
+ * Refusing *before* any column is read keeps a newer-format table from being
+ * silently mis-read as format 2 (or legacy). Two independent conditions:
+ * a `format` above what this build writes, or a `minReader` above the running
+ * engine version.
+ */
+export declare const unsupportedFormatReason: (format: TableFormat) => string | null;
+/**
+ * Read a table's format marker. A missing or unreadable marker is legacy
+ * (format 1), which keeps every pre-3.8 table byte-for-byte compatible.
+ */
+export declare const readFormat: (tablePath: string) => Promise<TableFormat>;
+/** The exact bytes of a format marker (shared by write and compaction commit). */
+export declare const serializeFormat: (format: TableFormat) => string;
+/**
+ * Write (or replace) the format marker durably and atomically (§9): stage the
+ * bytes in the table's `.tmp` dir, fsync the temp, rename it into place, then
+ * fsync the table directory. A crash can therefore leave either the old marker
+ * or the new one, never a partial file — a torn marker would read back as
+ * legacy and silently drop the log.
+ */
+export declare const writeFormat: (tablePath: string, format: TableFormat) => Promise<void>;
+/** Remove the marker (used when downgrading a table back to legacy). */
+export declare const removeFormat: (tablePath: string) => Promise<void>;

package/dist/format.js

@@ -0,0 +1,126 @@
+import { randomUUID } from "node:crypto";
+import { readFile, rename, unlink } from "node:fs/promises";
+import { join } from "node:path";
+import * as File from "./file.js";
+/**
+ * On-disk format marker for a table directory.
+ *
+ * Format 1 is the implicit legacy format (no `.format` file): base columns plus
+ * pagination only. Format 2 opts the table into the append-only update log
+ * written by `put` (see `updatelog.ts`).
+ *
+ * The marker is the single source of truth for "does this table have a log?".
+ * It also carries the compatibility contract read by a later 3.7.1 detector:
+ * `minReader` is the oldest engine that may open the table, `detector` the
+ * release that hard-fails instead of silently mis-reading. `baseGen` is the
+ * generation the base columns were last compacted at; it is bumped by
+ * compaction (3.9) so a stale in-memory overlay can detect that its base moved
+ * and rebuild instead of replaying frames that are already merged.
+ */
+export const FORMAT_FILE = ".format";
+export const LEGACY_FORMAT = {
+    format: 1,
+    log: false,
+    baseGen: 0,
+};
+export const CURRENT_FORMAT_VERSION = 2;
+export const ENGINE_NAME = "inibase";
+/** This build's engine version, compared against a marker's `minReader`. */
+export const ENGINE_VERSION = "3.8.0";
+/** Oldest reader that understands format 2. */
+export const MIN_READER = "3.8.0";
+/** Release expected to detect (and refuse) format 2 rather than mis-read it. */
+export const DETECTOR = "3.7.1";
+/** Compare dotted numeric versions: >0 when `a > b`, <0 when `a < b`, else 0. */
+export const compareVersions = (a, b) => {
+    const pa = a.split(".");
+    const pb = b.split(".");
+    for (let i = 0; i < Math.max(pa.length, pb.length); i++) {
+        const diff = (Number(pa[i]) || 0) - (Number(pb[i]) || 0);
+        if (diff !== 0)
+            return diff > 0 ? 1 : -1;
+    }
+    return 0;
+};
+/**
+ * Why this engine cannot open a marker, or null when it can.
+ *
+ * Refusing *before* any column is read keeps a newer-format table from being
+ * silently mis-read as format 2 (or legacy). Two independent conditions:
+ * a `format` above what this build writes, or a `minReader` above the running
+ * engine version.
+ */
+export const unsupportedFormatReason = (format) => {
+    if (format.format > CURRENT_FORMAT_VERSION)
+        return `format ${format.format} (this engine reads up to ${CURRENT_FORMAT_VERSION})`;
+    if (format.minReader && compareVersions(format.minReader, ENGINE_VERSION) > 0)
+        return `minReader ${format.minReader} (this engine is ${ENGINE_VERSION})`;
+    return null;
+};
+const formatPath = (tablePath) => join(tablePath, FORMAT_FILE);
+/**
+ * Read a table's format marker. A missing or unreadable marker is legacy
+ * (format 1), which keeps every pre-3.8 table byte-for-byte compatible.
+ */
+export const readFormat = async (tablePath) => {
+    if (!(await File.isExists(formatPath(tablePath))))
+        return { ...LEGACY_FORMAT };
+    try {
+        const raw = JSON.parse(await readFile(formatPath(tablePath), "utf8"));
+        return {
+            // Accept both the current `format` key and the earlier `version`
+            // spelling so a marker written by any 3.8 prerelease still opens.
+            format: Number(raw?.format ?? raw?.version) || LEGACY_FORMAT.format,
+            log: raw?.log === true,
+            baseGen: Number(raw?.baseGen) || 0,
+            minReader: typeof raw?.minReader === "string" ? raw.minReader : undefined,
+        };
+    }
+    catch {
+        // A corrupt marker means we cannot trust "no log": refusing to write
+        // over it is safer, but reading must still work, so treat as legacy and
+        // let the write path surface UNSUPPORTED_FORMAT.
+        return { ...LEGACY_FORMAT };
+    }
+};
+/** The exact bytes of a format marker (shared by write and compaction commit). */
+export const serializeFormat = (format) => `${JSON.stringify({
+    engine: ENGINE_NAME,
+    format: format.format,
+    minReader: MIN_READER,
+    detector: DETECTOR,
+    layout: "append",
+    log: format.log,
+    baseGen: format.baseGen,
+})}\n`;
+/**
+ * Write (or replace) the format marker durably and atomically (§9): stage the
+ * bytes in the table's `.tmp` dir, fsync the temp, rename it into place, then
+ * fsync the table directory. A crash can therefore leave either the old marker
+ * or the new one, never a partial file — a torn marker would read back as
+ * legacy and silently drop the log.
+ */
+export const writeFormat = async (tablePath, format) => {
+    const path = formatPath(tablePath);
+    const tmp = join(tablePath, ".tmp", `${FORMAT_FILE}.${randomUUID()}`);
+    try {
+        await File.write(tmp, serializeFormat(format));
+        await File.syncFile(tmp);
+        await rename(tmp, path);
+        await File.syncDir(tablePath);
+    }
+    catch (error) {
+        await unlink(tmp).catch(() => { });
+        throw error;
+    }
+};
+/** Remove the marker (used when downgrading a table back to legacy). */
+export const removeFormat = async (tablePath) => {
+    const path = formatPath(tablePath);
+    try {
+        await unlink(path);
+    }
+    catch {
+        // Already absent.
+    }
+};

package/dist/index.d.ts

@@ -44,6 +44,14 @@ export interface TableConfig {
     cache?: boolean;
     prepend?: boolean;
     decodeID?: boolean;
+    /**
+     * Opt a table into the incremental update log (format 2): `put` appends a
+     * small frame instead of rewriting whole columns, and reads merge the log
+     * over the base columns. Requires a schema of simple top-level fields and
+     * is incompatible with `prepend`, transactions and DDL on the table
+     * (those need compaction, shipped in 3.9).
+     */
+    log?: boolean;
 }
 export interface TableObject {
     schema?: Schema;
@@ -88,7 +96,7 @@ declare global {
         entries<T extends object>(o: T): Entries<T>;
     }
 }
-export declare const ERROR_CODES: readonly ["GROUP_UNIQUE", "FIELD_UNIQUE", "FIELD_REQUIRED", "NO_SCHEMA", "TABLE_EMPTY", "INVALID_ID", "INVALID_TYPE", "INVALID_PARAMETERS", "NO_ENV", "TABLE_EXISTS", "TABLE_NOT_EXISTS", "INVALID_REGEX_MATCH", "INVALID_NAME", "COMPUTED_FIELD_SYNTAX", "COMPUTED_FIELD_UNKNOWN_FIELD", "COMPUTED_FIELD_INVALID_LINK", "COMPUTED_FIELD_INVALID_TARGET", "COMPUTED_FIELD_CONFLICT", "COMPUTED_FIELD_CYCLE", "COMPUTED_FIELD_SETTABLE", "COMPUTED_FIELD_DANGLING_LINK", "COMPUTED_FIELD_ARITHMETIC"];
+export declare const ERROR_CODES: readonly ["GROUP_UNIQUE", "FIELD_UNIQUE", "FIELD_REQUIRED", "NO_SCHEMA", "TABLE_EMPTY", "INVALID_ID", "INVALID_TYPE", "INVALID_PARAMETERS", "NO_ENV", "TABLE_EXISTS", "TABLE_NOT_EXISTS", "INVALID_REGEX_MATCH", "INVALID_NAME", "COMPUTED_FIELD_SYNTAX", "COMPUTED_FIELD_UNKNOWN_FIELD", "COMPUTED_FIELD_INVALID_LINK", "COMPUTED_FIELD_INVALID_TARGET", "COMPUTED_FIELD_CONFLICT", "COMPUTED_FIELD_CYCLE", "COMPUTED_FIELD_SETTABLE", "COMPUTED_FIELD_DANGLING_LINK", "COMPUTED_FIELD_ARITHMETIC", "UPDATE_LOG_CORRUPT", "UPDATE_LOG_FRAME_TOO_LARGE", "MAINTENANCE_REQUIRED", "UNSUPPORTED_FORMAT"];
 export type ErrorCode = (typeof ERROR_CODES)[number];
 export type ErrorLang = "en" | "ar" | "fr" | "es";
 export declare const globalConfig: {
@@ -121,6 +129,15 @@ export default class Inibase {
      *  changes: see the invalidation in `getTable`'s reload branch,
      *  `createTable` and `updateTableLocked`. Bounded by the number of tables. */
     private readonly computedPlanCache;
+    /** Per-table `.format` marker + live update-log overlays (format 2 tables).
+     *  Keyed by `databasePath\0tableName` so two databases in one process never
+     *  share state. Populated lazily by `getOverlay`. */
+    private readonly tableFormats;
+    private readonly overlays;
+    /** Last observed identity of each table's `.format` file (`mtime:size:ino`),
+     *  used to detect a compaction by another process without re-reading the
+     *  marker on every access. */
+    private readonly formatStamps;
     private databasePath;
     private uniqueMap;
     private schemaFileExtension;
@@ -176,6 +193,74 @@ export default class Inibase {
      * @return {*}  {Promise<TableObject | undefined>}
      */
     getTable(tableName: string): Promise<TableObject | undefined>;
+    private overlayKey;
+    /** True when the table opted into the incremental update log. */
+    private isLogTable;
+    /**
+     * Resolve a table's format marker, revalidating a cached *log* table when
+     * its `.format` file changed underneath us. A compaction by another process
+     * bumps `baseGen` and empties the log; without re-reading the marker this
+     * process would scan the new (empty) log with the old `baseGen`, and once a
+     * post-compaction frame appears the generation gap would look like
+     * corruption. Legacy tables never change format implicitly, so they skip
+     * the stat entirely and keep the pre-3.8 read path cost.
+     */
+    /**
+     * Read a table's `.format` marker and refuse a table written by a newer
+     * engine (§4.1) *before* any column is read: a higher `format`, or a
+     * `minReader` above this build, throws UNSUPPORTED_FORMAT rather than being
+     * silently mis-read as format 2/legacy.
+     */
+    private readValidatedFormat;
+    private currentFormat;
+    /**
+     * Load (or refresh) a table's live overlay. Returns null for legacy tables
+     * (no `.format` marker) so every existing code path stays untouched. With
+     * `forWrite`, repairs a torn tail and rebuilds to the committed boundary;
+     * the caller must already hold the table writer lock.
+     */
+    private getOverlay;
+    private stampFormat;
+    /**
+     * The format-2 update log stores one encoded cell per changed *top-level
+     * simple* column; nested (object/array/table) and computed fields are not
+     * representable without a compaction step (3.9). A log table whose schema
+     * violates this is rejected at DDL time so the write path can always take
+     * the log and never needs to interleave a whole-column rewrite.
+     */
+    private assertLogCompatibleSchema;
+    private idFieldOf;
+    /** Read the internal numeric id for each requested line (stable overlay key). */
+    private overlayLineIds;
+    /** Internal id -> current line number(s), built from the id column. */
+    private overlayIdToLines;
+    /**
+     * Per-line raw-cell overrides for one column, from the overlay. Used by
+     * `File.search` so criteria, uniqueness and aggregate scans see the merged
+     * (un-compacted) values. Returns undefined when the table has no overlay or
+     * the column is never overlaid.
+     */
+    private overlayColumnOverrides;
+    /**
+     * Fold every logged update into the base columns and reset the log.
+     *
+     * Safe to call on any table: legacy tables (no `.format` marker) are a
+     * no-op. Semantically neutral — it only materializes already-acknowledged
+     * updates, so a failure never makes an acknowledged `put` disappear.
+     * Returns the new `baseGen` for a compacted table, or `null` when there was
+     * nothing to fold.
+     */
+    compact(tableName: string): Promise<number | null>;
+    /**
+     * Compaction core. The caller must hold the table writer lock. Streams each
+     * overlaid base column through `File.replace` (which handles compression),
+     * then publishes the new columns, an empty `updates.log`, and a `.format`
+     * marker with the bumped generation in one journaled commit, so a crash
+     * either rolls every file back (old base + old log + old `baseGen`) or
+     * forward (new base + empty log + new `baseGen`). No acknowledged update is
+     * ever lost.
+     */
+    private compactLogTable;
     getTableSchema(tableName: string): Promise<Schema>;
     private throwErrorIfTableEmpty;
     validateData(data: Data | Data[], schema: Schema, skipRequiredField?: boolean): void;
@@ -190,6 +275,17 @@ export default class Inibase {
     private joinPathesContents;
     private _processSchemaDataHelper;
     private processSchemaData;
+    /**
+     * Fold the table's un-compacted update-log overlay onto decoded rows.
+     *
+     * `RETURN` is keyed by line number and its `id` is the user-facing id, so
+     * the overlay (keyed by the internal numeric id) is matched through a fresh
+     * id-column read of just the requested lines. Only columns present in
+     * `schema` (which is already narrowed to the requested `columns`) are
+     * written, so a partial projection never grows extra keys. Legacy tables
+     * (no `.format` marker) return immediately.
+     */
+    private mergeOverlayIntoRows;
     /**
      * Extract the raw expression from a field's `computed` property (string or
      * persisted `{ expr, ast }` spec).
@@ -382,6 +478,13 @@ export default class Inibase {
     get<_TData extends Record<string, any> & Partial<Data>>(tableName: string, where: string | number | (string | number)[] | Criteria | undefined, options: Options | undefined, onlyOne: true, onlyLinesNumbers: true, _whereIsLinesNumbers?: boolean): Promise<number | null>;
     private _get;
     private getOnce;
+    /**
+     * In-process sort fallback for log tables with a non-empty overlay: the
+     * shell `paste | sort | awk` path reads base cells and would miss puts.
+     * Reads the merged rows for the (optionally filtered) lines, sorts them by
+     * the requested keys with type-aware comparison, then paginates.
+     */
+    private getOnceSortedWithOverlay;
     /**
      * Create new item(s) in a table
      *
@@ -414,6 +517,16 @@ export default class Inibase {
     put<TData extends Record<string, any> & Partial<Data>>(tableName: string, data: Data & TData, where: number | string | (number | string)[] | Criteria | undefined, options: Options | undefined, returnUpdatedData: true | boolean, _whereIsLinesNumbers?: boolean): Promise<(Data & TData) | null>;
     put<TData extends Record<string, any> & Partial<Data>>(tableName: string, data: (Data & TData)[], where: number | string | (number | string)[] | Criteria | undefined, options: Options | undefined, returnUpdatedData: true | boolean, _whereIsLinesNumbers?: boolean): Promise<(Data & TData)[] | null>;
     put<TData extends Record<string, any> & Partial<Data>>(tableName: string, data: (Data & TData) | (Data & TData)[], where: number | string | (number | string)[] | Criteria | undefined, options: Options | undefined, returnUpdatedData: true | boolean, _whereIsLinesNumbers?: boolean): Promise<(Data & TData) | (Data & TData)[] | null>;
+    /**
+     * Incremental `put` for format-2 (log) tables.
+     *
+     * Resolves the target rows to their stable internal ids, validates the
+     * payload against the merged view (uniqueness included), and appends one
+     * bounded frame instead of rewriting whole columns. The base files are left
+     * untouched; readers merge the overlay. Returns `undefined` when nothing
+     * matches, matching the legacy `put` contract.
+     */
+    private putViaLog;
     private _put;
     /**
      * Delete item(s) in a table