latticesql 1.4.0 → 1.6.0

package/README.md

@@ -1,6 +1,6 @@
 # latticesql
 
-**Persistent memory for AI agents.** Keeps a SQLite database and a set of context files in sync — so every agent session starts with accurate state, and agent output becomes permanent data.
+**Persistent memory for AI agents.** Keeps a SQLite **or Postgres** database and a set of context files in sync — so every agent session starts with accurate state, and agent output becomes permanent data.
 
 [![npm version](https://img.shields.io/npm/v/latticesql.svg)](https://www.npmjs.com/package/latticesql)
 [![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](./LICENSE)
@@ -59,6 +59,7 @@ Lattice has no opinions about your schema, your agents, or your file format. You
 - [CLI — lattice generate](#cli--lattice-generate)
 - [Schema migrations](#schema-migrations)
 - [Security](#security)
+- [Pluggable backends (v1.6+)](#pluggable-backends-v16)
 - [Architecture](#architecture)
 - [Examples](#examples)
 - [Staying up to date](#staying-up-to-date)
@@ -74,7 +75,24 @@ Lattice has no opinions about your schema, your agents, or your file format. You
 npm install latticesql
 ```
 
-Requires **Node.js 18+**. Uses `better-sqlite3` — no external database process needed.
+Requires **Node.js 18+**. The default backend is SQLite (`better-sqlite3`) — no external database process needed.
+
+To use the Postgres backend (for Supabase, Neon, RDS, or any other Postgres-compatible database), install the optional dependencies:
+
+```bash
+npm install latticesql pg synckit
+```
+
+Then pass a connection string instead of a file path:
+
+```ts
+import { Lattice } from 'latticesql';
+
+const lattice = new Lattice('postgres://user:pass@host:5432/db');
+// rest of your setup is identical to the SQLite path
+```
+
+See [Pluggable backends](#pluggable-backends-v16) below for full details.
 
 ---
 
@@ -861,22 +879,6 @@ Migrations are idempotent — each `version` number is applied exactly once, tra
 
 `close()` closes the SQLite connection. Call it when the process shuts down.
 
-#### Migration validation (v1.4+)
-
-Pass a `validateMigrationSQL` function in `InitOptions` to validate migration SQL before any migrations execute. If validation fails, no migrations run and an error is thrown.
-
-```typescript
-await db.init({
-  migrations: [{ version: 1, sql: 'ALTER TABLE tasks ADD COLUMN due_date TEXT' }],
-  validateMigrationSQL: (sql) => {
-    if (sql.trim().length === 0) return { valid: false, errors: ['Empty SQL'] };
-    return { valid: true };
-  },
-});
-```
-
-Multi-statement migrations are fully supported — each migration's SQL can contain multiple semicolon-separated statements, all of which are executed.
-
 ### `migrate()` (v0.17+)
 
 ```typescript
@@ -1068,30 +1070,6 @@ await db.count(table: string, opts?: CountOptions): Promise<number>
 const n = await db.count('tasks', { where: { status: 'open' } });
 ```
 
-#### `isDirty()` / `markDirty()` (v1.4+)
-
-```typescript
-db.isDirty(): boolean
-db.markDirty(table?: string): this
-```
-
-Lattice tracks a per-table write version counter. `isDirty()` returns `true` when any table has been written to since the last `render()` call — useful for consumers implementing custom polling loops to skip redundant render cycles.
-
-`markDirty()` forces one or all tables to be considered changed. Call it after writing directly via the `db` escape hatch.
-
-```typescript
-// Custom watch loop that skips redundant renders
-setInterval(async () => {
-  if (db.isDirty()) {
-    await db.render(outputDir);
-  }
-}, 5000);
-
-// After direct DB writes, mark dirty so next render picks up changes
-db.db.prepare('UPDATE tasks SET status = ? WHERE id = ?').run('done', taskId);
-db.markDirty('tasks');
-```
-
 ---
 
 ### Query operators
@@ -1304,18 +1282,6 @@ const rows = db.db
   .all('open');
 ```
 
-After direct writes via the escape hatch, call `db.markDirty('table')` so `isDirty()` reflects the change.
-
----
-
-### Performance (v1.4+)
-
-Lattice v1.4 includes two internal performance improvements that require no API changes:
-
-- **Prepared statement cache** — `SQLiteAdapter` caches compiled `better-sqlite3` statements. Repeated calls with the same SQL string reuse the compiled statement instead of recompiling. DDL statements bypass the cache. The cache clears automatically after schema changes and on `close()`.
-
-- **Batch entity query resolution** — Entity context rendering pre-fetches related rows for all entities in a single `WHERE IN (...)` query per source, replacing the previous per-entity query pattern. `hasMany`, `manyToMany`, and `belongsTo` sources are batched automatically. `custom` and `enriched` sources fall back to per-entity resolution.
-
 ---
 
 ### Context optimization (v1.3+)
@@ -2153,6 +2119,100 @@ const db = new Lattice('./app.db', {
 
 ---
 
+## Pluggable backends (v1.6+)
+
+Lattice ships with two storage adapters and a pluggable interface so you can bring your own.
+
+### Picking a backend by connection string
+
+The `Lattice` constructor inspects the first argument and picks the right adapter:
+
+| First argument | Adapter | When to use |
+|---|---|---|
+| `'/abs/path/to/db.sqlite'` (or any plain path) | `SQLiteAdapter` | Default. Local file, no server. |
+| `':memory:'` | `SQLiteAdapter` | In-memory SQLite. Great for tests. |
+| `'file:/abs/path/to/db.sqlite'` | `SQLiteAdapter` | Same as the plain path form, with the scheme spelled out. |
+| `'postgres://user:pass@host:5432/db'` | `PostgresAdapter` | Postgres-compatible cloud DB (Supabase, Neon, RDS, …). |
+| `'postgresql://user:pass@host:5432/db'` | `PostgresAdapter` | Same as `postgres://`. |
+| any string + `{ adapter: myAdapter }` | your adapter | Bring your own implementation. |
+
+```ts
+import { Lattice } from 'latticesql';
+
+// SQLite (default)
+const local = new Lattice('./data/lattice.db');
+
+// Postgres (Supabase / Neon / RDS / etc.)
+const cloud = new Lattice('postgres://postgres:secret@db.example.com:5432/agent');
+
+// Bring your own
+const custom = new Lattice('ignored', { adapter: new MyCustomAdapter() });
+```
+
+The rest of the API — `define()`, `init()`, `query()`, `insert()`, `render()`, `migrate()`, `watch()`, `reverseSync()`, `reverseSeed()` — is unchanged across both backends.
+
+### Postgres setup
+
+`PostgresAdapter` depends on `pg` and `synckit`. Both are listed as `optionalDependencies`, so SQLite-only consumers don't pay the install cost. Install them when you actually use Postgres:
+
+```bash
+npm install pg synckit
+```
+
+Then point Lattice at any Postgres-compatible database that speaks the standard wire protocol on port 5432:
+
+```ts
+const lattice = new Lattice('postgres://user:pass@host:5432/db');
+await lattice.init();
+```
+
+**Connection pooler note:** if you put a pooler (e.g. PgBouncer, Supabase pooler) in front of your database, prefer **session-mode pooling**. Transaction-mode poolers do not support prepared statements across transactions, which would break Lattice's `adapter.prepare()` pattern.
+
+**Why a worker thread under the hood:** the `StorageAdapter` interface is synchronous because `better-sqlite3` is sync. Every Node Postgres client is async. `PostgresAdapter` runs `pg` inside a `synckit` worker thread and blocks the main thread on `Atomics.wait` until the worker posts its reply. Each query pays ~1–3 ms of message-passing overhead — fine for Lattice's batch-insert + periodic-render workload. If you ever need OLTP-grade throughput, the interface can grow an async variant without breaking SQLite consumers.
+
+**Schema portability:** Lattice's table definitions are mostly portable SQL. The adapter handles the few dialect differences automatically:
+
+- `?` placeholders are translated to `$1, $2, …` for Postgres. Single-quoted strings, double-quoted identifiers, and SQL comments are skipped — `?` characters inside those are left alone.
+- `BLOB` column types are translated to `BYTEA` inside `addColumn`. Use `BLOB` in your `TableDefinition` and it works on both backends.
+- `datetime('now')` and `RANDOM()` defaults are translated to `NOW()` and `random()` for Postgres.
+- Use `TEXT PRIMARY KEY` (UUIDs) for portable primary keys. `INTEGER PRIMARY KEY` auto-increments on SQLite but not Postgres — if you need it on Postgres, use a sequence or `GENERATED ALWAYS AS IDENTITY`.
+
+### Bring your own adapter
+
+The interface is small enough to implement against any backend:
+
+```ts
+export interface StorageAdapter {
+  run(sql: string, params?: unknown[]): void;
+  get(sql: string, params?: unknown[]): Row | undefined;
+  all(sql: string, params?: unknown[]): Row[];
+  prepare(sql: string): PreparedStatement;
+  open(): void;
+  close(): void;
+  introspectColumns(table: string): string[];
+  addColumn(table: string, column: string, typeSpec: string): void;
+}
+```
+
+Pass your implementation via `options.adapter`:
+
+```ts
+import { Lattice } from 'latticesql';
+import type { StorageAdapter } from 'latticesql';
+
+class MyMySQLAdapter implements StorageAdapter { /* … */ }
+
+const lattice = new Lattice('ignored', { adapter: new MyMySQLAdapter() });
+```
+
+### Limitations
+
+- `PreparedStatement.run()` returns `lastInsertRowid: 0` on the Postgres path. SQLite consumers that rely on `lastInsertRowid` should switch to `TEXT PRIMARY KEY` (UUIDs) for portability, or write `INSERT … RETURNING id` queries explicitly.
+- Two SQLite-only paths remain: `fixSchemaConflicts(db)` (the lifecycle helper that takes a raw `Database.Database` argument) and the writeback session-apply machinery. Postgres consumers shouldn't call them.
+- A built-in migration tool (SQLite → Postgres) is not included. Use a generic SQLite → Postgres migration tool, or `INSERT … SELECT` row-by-row.
+
+---
+
 ## Architecture
 
 ```
@@ -2172,7 +2232,7 @@ const db = new Lattice('./app.db', {
 │                  │ entity contexts  │                               │
 ├──────────────────┴──────────────────┴───────────────────────────────┤
 │                         SQLiteAdapter                                │
-│       (better-sqlite3 — synchronous I/O, statement cache)           │
+│              (better-sqlite3 — synchronous I/O)                     │
 └──────────────────────────────────────────────────────────────────────┘
         │                              │
         │ compileRender()              │ lifecycle/
@@ -2190,7 +2250,7 @@ const db = new Lattice('./app.db', {
 
 **Key design decisions:**
 
-- **Synchronous SQLite** — `better-sqlite3` gives synchronous reads; all Lattice CRUD methods return Promises for API consistency but resolve synchronously under the hood. Prepared statements are cached and reused automatically (v1.4+).
+- **Synchronous SQLite** — `better-sqlite3` gives synchronous reads; all Lattice CRUD methods return Promises for API consistency but resolve synchronously under the hood.
 - **Compile-time render** — `RenderSpec` is compiled to a plain `(rows: Row[]) => string` function at `define()`-time, not at render-time. `RenderEngine` stays unchanged.
 - **Atomic writes** — files are written to a `.tmp` sibling then renamed. No partial writes, no reader sees incomplete content.
 - **Schema-additive only** — Lattice never drops tables or columns automatically; it only adds missing ones.