inibase 2.0.1 → 3.1.0

package/README.md

@@ -1,6 +1,6 @@
 # Inibase :pencil:
 
-> A file-based & memory-efficient, serverless, ACID compliant, relational database management system :fire:
+> A file-based & memory-efficient, serverless relational database with **crash-atomic ACID: single-table DML + multi-table transactions** :fire: — per-table writer locks, write-ahead journal + crash recovery, fsync-backed durability, and a `begin`/`commit`/`rollback` transaction API (exact scope in [Durability & crash safety](#durability--crash-safety)).
 
 [![Inibase banner](./.github/assets/banner.jpg)](https://github.com/inicontent/inibase)
 
@@ -12,7 +12,7 @@
 - **Minimalist** :white_circle: (but powerful)
 - **100% TypeScript** :large_blue_diamond:
 - **Super-Fast** :zap: (built-in caching system)
-- **ATOMIC** :lock: File lock for writing
+- **ATOMIC** :lock: Per-table writer locks, write-ahead journal + crash recovery, fsync-backed durability, and multi-table transactions (`begin`/`commit`/`rollback`) for atomic cascades (exact scope below)
 - **Built-in** form validation (+unique values :new: ) :sunglasses:
 - **Suitable for large data** :page_with_curl: (tested with 4M records)
 - **Support Compression** :eight_spoked_asterisk: (using built-in nodejs zlib)
@@ -75,6 +75,98 @@ const users = await db.get("user", { favoriteFoods: "![]Pizza,Burger" });
 
 This structure ensures efficient storage, retrieval, and updates, making our system scalable and high-performing for diverse datasets and applications.
 
+## Durability & crash safety
+
+> [!IMPORTANT]
+> **DML (post / put / delete / truncate) is crash-atomic per table.** A logical
+> operation spans many column files plus a pagination-metadata file; all of
+> those files are committed as one journaled unit, so a table is never observed
+> half-applied. **Multi-table atomicity** is available explicitly through the
+> transaction API (`begin` / `commit` / `rollback`, see below) — cascade
+> deletes and multi-table writes are atomic *inside* a transaction, and
+> best-effort without one.
+
+### How each property is provided
+
+| Property | Mechanism |
+|---|---|
+| **A**tomicity | Write-ahead journal: one journal per table (`.tmp/journal.jsonl`) for single-table DML, plus a database journal (`<db>/.tmp/journal.jsonl`) that transactions append `op` entries to. Intent is fsynced *before* any live file changes; a `commit` marker is fsynced after publication. On crash: no commit marker → roll back (restore backups), commit present → roll forward (complete swaps, discard backups). Recovery runs automatically under the lock before any mutation/read. |
+| **C**onsistency | Schema validation + uniqueness enforcement happen before anything is written; every operation is atomic, so a table is never observed half-applied. |
+| **I**solation | **One writer lock per table** serializes every mutation (single-host and multi-host / NFS); a transaction holds the writer lock of every table it touches for its whole lifetime, plus a database lock to serialize transactions. Reads are **lock-free** with optimistic version-retry: each read snapshots the identity of every column + pagination file, and is re-run if anything changed mid-scan — a reader never sees torn rows. |
+| **D**urability | Durability knob `INIBASE_DURABILITY=full` (default) fsyncs the temp file, the journal (begin/commit), and the affected directories (incl. shell `sed`/`gzip` temp paths and stream pipelines). DDL (create/update table, compression & prepend toggles) is fsynced too. `INIBASE_DURABILITY=none` skips **every** fsync while keeping the exact journal protocol — the data is process-crash safe (a crash leaves an in-flight journal that recovery rolls back or forward) but **not** power-loss safe, because the OS page cache can be lost. |
+
+**Commit-point ordering:** the pagination metadata rename happens *first* (the atomic publication point — the row count flips in a single rename that lock-free readers observe), then column files are swapped `live → backup` + `tmp → live`.
+
+### Transactions (multi-table atomicity)
+
+```js
+await db.begin(["orders", "invoices"]); // pre-lock tables in sorted order (deadlock-free)
+try {
+  await db.post("orders", order);
+  await db.post("invoices", invoice);
+  await db.commit(); // publish everything crash-atomically
+} catch {
+  await db.rollback(); // discard everything; no live file was touched
+}
+```
+
+- **Semantics:** mutations issued inside a transaction are *staged* — their
+  temps are fsynced and an `op` entry appended to the database journal — but no
+  live file changes until `commit()`. `commit()` publishes every staged
+  mutation (per table: pagination rename first, then column swaps) and ends
+  with a single fsynced `commit` marker, making the whole set atomic with
+  respect to crashes and process kills. `rollback()` removes the temps and the
+  journal without touching any live file.
+- **Cascade:** a `delete` inside a transaction cascades into referencing tables
+  *through the same journal*, so the whole parent + children removal is atomic;
+  a failure during the cascade aborts the transaction.
+- **Rules & limits (v1):**
+  - **No read-your-writes:** reads inside a transaction observe the last
+    committed state, not staged writes.
+  - **One mutation per table per transaction.** A second post/put/delete on a
+    table already staged in the same transaction throws `INVALID_PARAMETERS`
+    (composing same-table writes would require read-your-writes).
+  - **DDL is not allowed inside a transaction** (`createTable` / `updateTable`
+    throw `INVALID_PARAMETERS`).
+  - `begin` without a table list locks tables on first touch, in first-touch
+    order; list the tables up front to get sorted, cycle-free ordering.
+  - `returnPostedData` inside a transaction returns the formatted staged rows
+    (no joins); `returnUpdatedData` inside a transaction throws
+    `INVALID_PARAMETERS`.
+  - Transactions are excluded from structural lock-upgrade deadlocks, but two
+    long transactions that touch overlapping tables can only deadlock in the
+    first-touch-order case (pre-listing avoids it).
+
+### Scope & caveats
+
+- **Best-effort without a transaction.** Cross-table cascade deletes outside a
+  transaction stay best-effort: referencing rows are removed *after* the
+  primary delete commits, and a crash in between can leave orphaned references.
+  Wrap them in `begin`/`commit` for atomicity.
+- **NFS:** lock files use `O_EXCL` creation, which is advisory on some NFS
+  servers — two hosts may briefly both believe they hold a lock. Locks store
+  `{pid, host, startedAt}`: a same-host owner that is **provably dead**
+  (`kill(pid, 0)` fails) is stolen immediately; foreign/unknown owners fall
+  back to the TTL (`INIBASE_LOCK_TTL_MS`, default `60000`).
+- **fsync guarantees:** durability assumes the OS/filesystem honours `fsync`.
+  Some disks and virtualized filesystems silently ignore it; directory `fsync`
+  is unsupported on a few platforms (best-effort there). `none` mode
+  deliberately gives up power-loss durability — that is the whole point of the
+  knob.
+- **Cost:** at `full`, every mutation fsyncs the temp file, journal, and
+  directories — expect noticeably slower hot-path writes than pre-durability
+  builds (the benchmark compares `full` vs `none` side by side). The swap
+  protocol also transiently holds temp + backup + live copies (~2–3× a
+  column's size; worst case is a `put` with no `where` — a full-table rewrite).
+- **Cache (`.cache`):** entries are derived, rebuildable artifacts versioned by
+  the row count; they detect staleness but never participate in the ACID
+  guarantee. Non-fsync'd.
+
+Run `pnpm test:durability` for the journal-recovery, multi-process writer,
+live-reader and stale-lock test suite, `pnpm test:transaction` for the
+transaction/cascade/crash-recovery suite, and `pnpm benchmark:durability` for
+the `full` vs `none` throughput comparison.
+
 ## Inibase CLI
 
 ```shell
@@ -766,6 +858,97 @@ await db.get("user", undefined, { sort: {age: -1, username: "asc"} });
 </blockquote>
 </details>
 
+<details>
+<summary>Computed Fields</summary>
+<blockquote>
+
+A `computed` column is derived from other columns of the same row whenever the
+row is written (`post` / `put`, including backfill on `updateTable`). The
+result is stored in its own column file like any other value, so reads and
+criteria queries work unchanged.
+
+```ts
+const db = new Inibase("/databaseName");
+
+await db.createTable("product", [
+	{ key: "name", type: "string" },
+	{ key: "price", type: "number" },
+]);
+
+// ids are assigned per table in schema order: customer=1, status=2, items=3,
+// product=4, quantity=5, unitPriceCents=6, totalCents=7, totalCentsLive=8
+await db.createTable("orders", [
+	{ key: "customer", type: "string" },
+	{ key: "status", type: "number" },
+	{
+		key: "items",
+		type: "array",
+		children: [
+			{ key: "product", type: "table", table: "product" },
+			{ key: "quantity", type: "number" },
+			{ key: "unitPriceCents", type: "number" },
+		],
+	},
+	{ key: "totalCents", type: "number", computed: "sum(5, 6)" }, // quantity x unitPriceCents
+	{ key: "totalCentsLive", type: "number", computed: "sum(5, 4.2)" }, // quantity x product.price
+]);
+
+const posted = await db.post(
+	"orders",
+	{
+		customer: "acme",
+		status: 1,
+		items: [
+			{ product: "id-of-widget", quantity: 2, unitPriceCents: 250 },
+			{ product: "id-of-gadget", quantity: 1, unitPriceCents: 100 },
+		],
+	},
+	undefined,
+	true,
+);
+// posted.totalCents === 600      (2*250 + 1*100)
+// posted.totalCentsLive === 847  (2*price(widget) + 1*price(gadget))
+```
+
+**Expression language (v1, integer-only).**
+
+- Operators: `+` `-` `,` (multiply) `/` `%`; `( )` for grouping. Multiplication
+  binds tighter than addition (`1 , 2 + 3` = `(1*2) + 3`).
+- Helpers: `sum` `count` `avg` `min` `max` iterate an array-of-objects found by
+  the ids inside the parentheses (`sum(5, 6)` = quantity x unit-price per item).
+- Paths: `id ("." id)*` where `.` hops through a `table` link
+  (`4.2` = price of the linked product row).
+- Integers only: there are no decimal literals — `.` is the path separator, so
+  `3.14` is a path (`field 3`, hop `field 14`), never 3.14. Fractional results
+  are written as division: `314 / 100` → 3.14.
+- A bare integer that matches a field id in the table's schema is that field
+  (ids are locative); a bare integer that matches no field is a literal.
+
+**Rules & limits (v1).**
+
+- Computed fields cannot be `required`, `unique` or `regex` (a
+  `COMPUTED_FIELD_CONFLICT` error).
+- Computed values are never user-settable (`post`/`put` with a computed key
+  throws `COMPUTED_FIELD_SETTABLE`).
+- Values are evaluated at write time and persisted; compiled expressions are
+  stored as `{ expr, ast }` in the schema. The AST carries only field ids, so
+  renaming a column (or its linked table's columns) never retargets an
+  expression.
+- `updateTable` backfills existing rows when a computed field is added or its
+  expression changes; a backfill that fails (missing dependency, arithmetic
+  error, dangling link) aborts the migration and leaves the schema untouched.
+- Aggregates: `sum` over no values is `0`, `count` is the element count, and
+  `avg`/`min`/`max` over no values throw `COMPUTED_FIELD_ARITHMETIC`.
+- Non-numeric operands, division/modulo by zero, and arithmetic over missing
+  (null) values throw `COMPUTED_FIELD_ARITHMETIC`; a link to a missing row
+  throws `COMPUTED_FIELD_DANGLING_LINK`.
+- Only top-level schema fields can be computed in v1; array contents are
+  reachable through helpers. Missing link values make a single bare path
+  evaluate to null (stored empty).
+
+</blockquote>
+</details>
+
 </blockquote>
 </details>
 
@@ -793,6 +976,31 @@ await db.get("user", undefined, { sort: {age: -1, username: "asc"} });
 > Default testing uses a table with username, email, and password fields, ensuring password encryption is included in the process<br>
 > Results are measured on a default table plus dedicated tables with `prepend`, `compression`, and `decodeID` configs enabled<br>
 > To run benchmarks, install _typescript_ & _[tsx](https://github.com/privatenumber/tsx)_ globally and run `benchmark` by default bulk, for single use `benchmark --single|-s`
+>
+> > [!WARNING]
+> > The numbers above were measured **before** always-on fsync + write-ahead journaling landed (they no longer reflect current hot-path write costs). Run `pnpm benchmark:durability` for the crash-atomic numbers.
+
+### Computed fields
+
+Write-time evaluation cost (helpers `sum(3 , 4)`, `avg(4)`, `min(4)`, `max(4)`, `count(3)` over an `items` array with 3 lines, plus an arithmetic field `itemTotal × (314 / 100)`), compared against the identical table without computed fields:
+
+| rows | POST bulk (plain / computed) | POST single (plain / computed) | PUT recompute (plain / computed) |
+|------|------------------------------|--------------------------------|----------------------------------|
+| 10   | 21.90 / 22.69 ms             | 25.24 / 26.80 ms               | 23.10 / 37.07 ms                 |
+| 100  | 22.00 / 25.23 ms             | 25.24 / 26.80 ms               | 22.16 / 40.00 ms                 |
+| 1000 | 28.79 / 35.15 ms             | 25.24 / 26.80 ms               | 21.01 / 45.05 ms                 |
+
+> Min of 3 rounds (fsync + journal on); the plain-vs-computed delta is the pure expression-evaluation cost (~6 ms/1000 rows of helpers on POST, more on `PUT` because every matched row is recomputed). GET all (1110 rows): 11.44 / 15.90 ms — computed values are stored in real column files, so reads never evaluate; the gap is the larger column count to scan. Run `pnpm benchmark:computed` to reproduce.
+
+**Link-heavy: shared product catalog with batched link-hop reads.** Every order's line items reference a shared 20-row catalog; `totalCentsLive = sum(quantity, product.price)` follows one link hop per line item. The engine batches the hops across the whole post: each batch resolves at most one read per **distinct** `(table, column, id)` triple (here: catalogSize=20 product rows) instead of one full row-level read per line item (2000 reads for 1000 orders × 2 items):
+
+| rows | POST bulk (lk link hop) |
+|------|-------------------------|
+| 10   | 42.29 ms                |
+| 100  | 47.76 ms                |
+| 1000 | 59.46 ms                |
+
+> Shared 20-row catalog (10/100/1000 orders × 2 line items). Without batching every line item re-reads its product row; with it each batch resolves ≤ catalogSize distinct linked rows once (deduplicated across every line item and every order of the batch) and re-evaluates against the warm in-memory cache. A dangling link anywhere in the batch still rejects the whole post (`COMPUTED_FIELD_DANGLING_LINK`). HTTP-request bound, not disk-bound: the cost scales with the number of *distinct* linked rows the catalog actually has, not with line-item count.
 
 ## Roadmap
 
@@ -835,6 +1043,7 @@ await db.get("user", undefined, { sort: {age: -1, username: "asc"} });
   - [ ] Encryption
   - [x] Data Compression
   - [x] Caching System
+  - [x] Computed fields (v1 id-only expression language)
   - [ ] Suggest [new feature +](https://github.com/inicontent/inibase/discussions/new?category=ideas)
 
 ## License

package/dist/expression.d.ts

@@ -0,0 +1,159 @@
+/**
+ * Computed-fields expression language (v1 — id-only).
+ *
+ * Grammar
+ * -------
+ * ```
+ * expression := term (("+" | "-") term)*
+ * term       := factor (("," | "/" | "%") factor)*   // "," = multiply
+ * factor     := integer-literal | path | function-call | "(" expression ")"
+ * path       := id ( "." id )*                       // "." = link/binding hop
+ * function   := "sum" | "count" | "avg" | "min" | "max" "(" expression ")"
+ * ```
+ *
+ * Every symbol is a numeric field `id` (per-table dense counter, nested
+ * children included). There are no decimal literals: `.` is reserved as the
+ * path separator, so `3.4` is a link hop (`field 3`, then `field 4` in the
+ * table `field 3` links to), never the decimal 3.4. Fractional results are
+ * reachable via division (`314 / 100` → 3.14).
+ *
+ * A bare integer that **matches a field id in the table's schema** is that
+ * field's value (ids are locative); a bare integer that matches no field id
+ * is an integer literal. This is what makes both `sum(4, 3.4)` (id 4 =
+ * quantity) and `314 / 100` (no such ids) usable.
+ *
+ * Compiled expressions are persisted with the schema as
+ * `{ expr: string, ast: CompiledExpressionNode }` so key/table renames never
+ * retarget an expression: the AST carries only field ids (+ the id of the
+ * array ancestor a helper iterates), and every key path is re-derived from
+ * the *current* schema at write time.
+ */
+import type { ErrorLang, Field, Schema } from "./index.js";
+/** Cap on the raw `computed` string length (bound work at validation). */
+export declare const COMPUTED_EXPR_MAX_LENGTH = 512;
+/** Cap on the parsed AST depth (protects the parser/evaluator recursion). */
+export declare const COMPUTED_EXPR_MAX_DEPTH = 64;
+export type ComputedFunctionName = "sum" | "count" | "avg" | "min" | "max";
+export type BinaryOp = "add" | "sub" | "mul" | "div" | "mod";
+/** Raw AST produced by {@link parseExpression} (field ids unresolved). */
+export type RawExpressionNode = {
+    kind: "num";
+    value: number;
+} | {
+    kind: "path";
+    ids: number[];
+} | {
+    kind: "bin";
+    op: BinaryOp;
+    left: RawExpressionNode;
+    right: RawExpressionNode;
+} | {
+    kind: "fn";
+    name: ComputedFunctionName;
+    arg: RawExpressionNode;
+};
+/**
+ * Compiled AST stored in the schema. Path nodes carry only ids (+ the id of
+ * the array ancestor a helper iterates); all key paths are resolved against
+ * the current schema at evaluation time, so renames never break expressions.
+ */
+export type CompiledExpressionNode = {
+    kind: "num";
+    value: number;
+} | {
+    kind: "path";
+    ids: number[];
+    arrayFieldId: number | null;
+} | {
+    kind: "bin";
+    op: BinaryOp;
+    left: CompiledExpressionNode;
+    right: CompiledExpressionNode;
+} | {
+    kind: "fn";
+    name: ComputedFunctionName;
+    /** id of the array ancestor every path in `arg` lives in. */
+    arrayFieldId: number;
+    arg: CompiledExpressionNode;
+};
+/** Persisted form of a `computed` schema property. (A type alias so it stays
+ *  assignable to Inison's recursive `Data` type.) */
+export type ComputedFieldSpec = {
+    expr: string;
+    ast: CompiledExpressionNode;
+};
+/**
+ * Schema index used for id resolution: maps a field id to its dotted key path,
+ * the resolved field, and the nearest array-of-objects ancestor (if any).
+ */
+export interface FieldRef {
+    key: string;
+    field: Field;
+    /** Nearest `array`-typed ancestor with object children, or null. */
+    arrayAncestor: {
+        id: number;
+        key: string;
+    } | null;
+    /** True when `arrayAncestor` itself sits inside another array of objects. */
+    nestedInArrayOfArrays: boolean;
+}
+/**
+ * Parse a `computed` expression into a raw AST. Throws
+ * `COMPUTED_FIELD_SYNTAX` on invalid syntax, oversized input or excessive
+ * nesting.
+ */
+export declare function parseExpression(source: string, language?: ErrorLang, fieldKey?: string): RawExpressionNode;
+/**
+ * Build the id → {@link FieldRef} index of a schema (nested children included,
+ * using dotted key paths). Container fields (array/object with object
+ * children) are indexed too so unknown ids are detected, but they can't be
+ * referenced by an expression.
+ */
+export declare function buildFieldIndex(schema: Schema): Map<number, FieldRef>;
+export interface ResolveContext {
+    language: ErrorLang;
+    /** Key of the computed field being compiled (error context). */
+    ownKey: string;
+    /** Id index of the table the computed field belongs to. */
+    index: Map<number, FieldRef>;
+    /** Fetch (and cache) the id index of another table, or undefined. */
+    getTableIndex: (tableName: string) => Promise<Map<number, FieldRef> | undefined>;
+}
+/**
+ * Resolve a raw expression against the schema (and linked tables), returning
+ * the compiled node plus the set of field ids it reads from the current
+ * table (used for dependency ordering / cycle detection between computed
+ * fields).
+ */
+export declare function resolveExpression(expression: RawExpressionNode, ctx: ResolveContext): Promise<{
+    ast: CompiledExpressionNode;
+    deps: Set<number>;
+}>;
+/** Every field id a compiled expression reads from the *current* table. */
+export declare function collectFieldDeps(node: CompiledExpressionNode): Set<number>;
+export interface ComputedFieldMeta {
+    id: number;
+    key: string;
+    deps: Set<number>;
+}
+/**
+ * Order computed fields so every dependency is evaluated before its dependents.
+ * Throws `COMPUTED_FIELD_CYCLE` when a field (transitively) depends on itself.
+ */
+export declare function topoSortComputedFields(fields: ComputedFieldMeta[], language: ErrorLang): ComputedFieldMeta[];
+/**
+ * Flatten a formatted row into a dot-notation record. Objects are flattened
+ * (`meta.age`), arrays of objects are combined per child key
+ * (`items.quantity` -> array of per-element values) and everything else is
+ * stored under its dotted key.
+ */
+export declare function flattenRecord(obj: Record<string, any>, prefix?: string): Record<string, any>;
+/**
+ * Resolve a dotted key path against a structured frame in place — the direct
+ * read-path counterpart of `flattenRecord`. Walks `obj` segment by segment
+ * and returns `undefined` when any intermediate is null, undefined, a
+ * non-object, or an array (arrays resolve numerically only, so they never
+ * match a dotted segment), mirroring the leaves `flattenRecord` would have
+ * produced without materialising the flattened record.
+ */
+export declare function resolveFramePath(obj: Record<string, any>, dottedKey: string): any;