latticesql 3.4.6 → 4.0.0

package/docs/consistency.md

@@ -0,0 +1,117 @@
+# Data Consistency
+
+How `latticesql` keeps the database, the rendered context tree, and offline edits
+in agreement — the true model, the invariants the library guarantees (each backed
+by a test), and the things it deliberately does **not** guarantee.
+
+---
+
+## The model
+
+There are three places data can live, and exactly one relationship between each
+pair:
+
+1. **The database** — the single source of truth. One active substrate at a time:
+   SQLite **xor** Postgres, never both. Every read and write goes here first.
+
+2. **The rendered context tree** — a **forward-render mirror** of the database
+   (`db → files`). It is derived, not authoritative: a render reads the DB and
+   writes Markdown files plus a manifest (`.lattice/manifest.json`) that records,
+   per file, the content hash and the source row's version. The tree **lags** the
+   DB until a render settles; it is never read back as truth except through the
+   explicit reverse-sync path below.
+
+3. **Offline edits** — a **client-side queue** (not a live mirror of the DB). Edits
+   made while disconnected are buffered and replayed, in order, when connectivity
+   returns.
+
+Everything below is about the three edges between these: DB ⇄ rendered tree, and
+DB ⇄ offline queue.
+
+---
+
+## Guaranteed invariants (each test-backed)
+
+### 1. Reverse-sync never silently overwrites a concurrent change
+
+When an external edit to a rendered file is swept back into the database, the
+engine first checks whether the underlying row changed since that file was
+rendered — an optimistic-concurrency check against the row version captured in the
+manifest at render time. If the row changed (a concurrent DB edit), the file edit
+is **rejected and reported as a conflict**, never applied over the newer value.
+The reject is surfaced to the editor so the change can be re-applied against the
+current record.
+
+_Why it matters:_ a file edit and a concurrent database edit to the same record
+can no longer race to a silent data loss — the database wins and the conflict is
+made visible.
+
+### 2. A render is manifest-atomic and fails loudly
+
+The manifest is written **last**, as a single atomic file (temp + rename). It is
+the commit point: a render either completes and commits a manifest describing a
+fully-written tree, or it throws **before** committing — leaving the **prior**
+manifest as the truthful record. Before writing anything, a render probes that its
+target directories are writable (a disk-full or read-only mount throws _before_ a
+single live file is touched). A write failure mid-render is re-raised loudly, never
+swallowed; the next render reconciles (unchanged files are skipped, and orphan
+cleanup runs only against a committed manifest).
+
+_Guarantee level:_ **manifest-atomic + tree-eventually-consistent.** A render is
+not a single atomic multi-file swap (a file tree cannot offer one without orphaning
+user-edited and attached files the tree interleaves), but the manifest — the one
+unit that must be atomic for correctness — is, and a failed render is self-healing
+rather than silently divergent.
+
+### 3. Render concurrency is single-owner per output directory
+
+Within a process, the auto-render scheduler and a background render share one
+in-flight guard, so they never overlap on the same output directory; a render
+scheduled while one is in flight is deferred and coalesced. One-shot CLI renders
+construct their own instance and do not auto-render, so they cannot interleave
+either.
+
+### 4. Migration reports what it leaves behind
+
+Migrating a workspace into a fresh database asserts, per table, that the target row
+count matches the source after copy (a mismatch aborts loudly, before the source is
+archived). Files whose canonical bytes are owned-and-local — and therefore are not
+carried by a row copy — are counted and reported, so the operator is told exactly
+how many files reference bytes that were left behind rather than discovering dangling
+references later.
+
+### 5. Offline replay is idempotent and ordered
+
+Queued edits carry a stable edit id and a client timestamp. Replay deduplicates by
+edit id and applies in timestamp order, so a replay that is retried (or partially
+re-sent) converges to the same state rather than double-applying.
+
+---
+
+## What is deliberately NOT guaranteed
+
+- **Full render-tree atomicity.** See invariant 2 — the manifest is atomic; the
+  surrounding file tree is eventually-consistent and self-healing, not
+  all-or-nothing.
+
+- **Multi-process renders to the same output directory.** Two processes rendering
+  the same context directory at once (for example, a long-running server plus a
+  separate one-shot render of the same directory) is **unsupported** — the manifest
+  write is last-writer-wins across processes. Point each long-running renderer at
+  its own output directory.
+
+- **Cross-key encryption round-trips.** Encrypted columns round-trip through
+  decrypt-on-read / encrypt-on-write; both sides of a migration must share the same
+  encryption key, or encrypted values arrive unreadable.
+
+---
+
+## Where each invariant is enforced
+
+| Invariant                                      | Enforced in                                                             |
+| ---------------------------------------------- | ----------------------------------------------------------------------- |
+| 1 — reverse-sync conflict gate                 | `src/reverse-sync/engine.ts` (row-version check), manifest `rowVersion` |
+| 2 — manifest-atomic render + writability probe | `src/render/engine.ts`, `src/render/writer.ts`                          |
+| 3 — single-owner render concurrency            | `src/render/auto-render.ts` (single-flight)                             |
+| 4 — migration row-count + blob surfacing       | `src/framework/cloud-migration.ts`                                      |
+| 5 — offline replay idempotency                 | the edit-id dedup + client-timestamp ordering on replay                 |

package/docs/examples/agent-system.md

@@ -60,9 +60,11 @@ entities:
       description: { type: text }
       status: { type: text, default: pending }
       priority: { type: integer, default: 1 }
-      agent_id: { type: uuid, ref: agent }
+      agent_id: { type: uuid }
       created_at: { type: datetime }
       completed_at: { type: datetime }
+    relations:
+      agent: { type: belongsTo, table: agent, foreignKey: agent_id }
     render:
       template: default-list
       formatRow: '[{{status}}] {{title}} → {{agent.name}}'
@@ -72,9 +74,11 @@ entities:
     fields:
       id: { type: uuid, primaryKey: true }
       type: { type: text, required: true }
-      agent_id: { type: uuid, ref: agent }
+      agent_id: { type: uuid }
       payload: { type: text }
       created_at: { type: datetime }
+    relations:
+      agent: { type: belongsTo, table: agent, foreignKey: agent_id }
     render:
       template: default-list
       formatRow: '{{created_at}} [{{type}}] {{agent.name}}'

package/docs/examples/cms.md

@@ -70,10 +70,12 @@ entities:
       title: { type: text, required: true }
       excerpt: { type: text }
       status: { type: text, default: draft }
-      author_id: { type: uuid, ref: author }
+      author_id: { type: uuid }
       word_count: { type: integer, default: 0 }
       published_at: { type: datetime }
       updated_at: { type: datetime }
+    relations:
+      author: { type: belongsTo, table: author, foreignKey: author_id }
     render:
       template: default-detail
       formatRow: '**{{title}}** [{{status}}] by {{author.name}} — {{word_count}} words'

package/docs/examples/ticket-tracker.md

@@ -59,10 +59,13 @@ entities:
       description: { type: text }
       status: { type: text, default: open }
       priority: { type: integer, default: 2 }
-      reporter_id: { type: uuid, ref: user }
-      assignee_id: { type: uuid, ref: user }
+      reporter_id: { type: uuid }
+      assignee_id: { type: uuid }
       created_at: { type: datetime }
       closed_at: { type: datetime }
+    relations:
+      reporter: { type: belongsTo, table: user, foreignKey: reporter_id }
+      assignee: { type: belongsTo, table: user, foreignKey: assignee_id }
     render:
       template: default-list
       formatRow: '[{{status}}] P{{priority}} {{title}} — {{assignee.name}}'
@@ -72,9 +75,12 @@ entities:
     fields:
       id: { type: uuid, primaryKey: true }
       body: { type: text, required: true }
-      ticket_id: { type: uuid, ref: ticket }
-      author_id: { type: uuid, ref: user }
+      ticket_id: { type: uuid }
+      author_id: { type: uuid }
       created_at: { type: datetime }
+    relations:
+      ticket: { type: belongsTo, table: ticket, foreignKey: ticket_id }
+      author: { type: belongsTo, table: user, foreignKey: author_id }
     render:
       template: default-list
       formatRow: '{{author.name}} ({{created_at}}): {{body}}'

package/docs/scaling.md

@@ -0,0 +1,56 @@
+# Scaling: connection pooling & bounded reads
+
+This note documents how `latticesql` behaves under concurrency — the Postgres
+connection-pool contract and the read-bounding posture — for deployments serving many
+simultaneous cloud users. It is the recorded outcome of the 4.0 cloud-scale review.
+
+## Connection pooling
+
+- **One pool per `Lattice` instance.** The Postgres adapter creates a single
+  `pg.Pool` with `max = poolSize` (default **10**). Configure it per instance:
+
+  ```ts
+  const db = new Lattice('postgres://…', { poolSize: 20 });
+  ```
+
+- **The data path reuses a long-lived instance, not a per-request one.** A GUI
+  workspace opens exactly one `Lattice` (cached as the active DB) and serves every
+  request for that workspace from its pool. The only short-lived instances are
+  **transient probes** during connect/credential checks (a workspace open's peek and
+  the connection-test probe) — they are not on the per-request data path and are
+  disposed promptly, so there is no per-request connection churn.
+
+- **The multi-tenant model is one connection identity per member.** Each cloud
+  member's GUI connects to the shared Postgres **as that member's own role** through
+  its own `Lattice`/pool. "Hundreds of concurrent members" therefore means hundreds of
+  independent role-scoped connections governed by Postgres `max_connections` (and any
+  external pooler such as PgBouncer), **not** contention on a single shared application
+  pool. Size `max_connections` / the pooler for the expected concurrent-member count;
+  keep each instance's `poolSize` modest (the default 10 is appropriate for a single
+  workspace's request concurrency).
+
+- **Statements are parameterized** (`pool.query(sql, params)`), so the driver reuses
+  prepared statements per connection. The one deliberate exception is the member-grant
+  reconcile, which uses the unparameterized simple-query protocol to batch a table's
+  GRANTs into a single round-trip (see `cloud/member-access.ts`).
+
+## Bounded reads (Rule of thumb: no unbounded whole-table read on a hot path)
+
+The main GUI list endpoints are bounded at the route layer via `parsePageParam` /
+`MAX_ROWS_PAGE` (`/api/tables/:t/rows`, `/api/system-tables/:t/rows`). The read API
+(`getActive`, `queryTable`) accepts optional `{ limit, offset }` bounds.
+
+A sweep of the remaining whole-table reads (empty-filter `query`/`queryTable`) found
+only these, each acceptable for the reason given — **none scale with row count on a
+per-request path**:
+
+| Site                                                                            | Why it is acceptable                                                                                                                                    |
+| ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `read-routes.ts` GUI metadata (`_lattice_gui_meta`, `_lattice_gui_column_meta`) | Bounded by **schema** size (≈ one row per entity/column), not data volume.                                                                              |
+| `reverse-seed/engine.ts`                                                        | Recovery path — runs only against an **empty / missing** entity table.                                                                                  |
+| `reverse-sync/engine.ts`                                                        | Inherent: diffs the full entity set against the rendered files; **debounced**, not per-request.                                                         |
+| `dedup-service.ts`                                                              | Inherent: duplicate detection must scan the candidate set. **Known limitation** for very large tables — a future windowed/indexed dedup would bound it. |
+
+When adding a read in a request handler, scheduler, or per-item loop, filter + bound it
+in SQL; if a whole-table read is genuinely required, justify it in a comment like the
+ones above so the next sweep can sign it off quickly.

package/docs/templates.md

@@ -267,7 +267,8 @@ When rendering, Lattice:
 1. Executes `SELECT * FROM users WHERE id = assignee_id` for each ticket
 2. Makes all `users` columns available as `{{assignee.<column>}}`
 
-In YAML config, `ref: user` automatically creates the `belongsTo` relation:
+In YAML config, declare the `belongsTo` relation in an entity-level `relations:`
+block (the relation name — `assignee` here — is what you reference in the template):
 
 ```yaml
 entities:
@@ -275,7 +276,9 @@ entities:
     fields:
       id: { type: uuid, primaryKey: true }
       title: { type: text, required: true }
-      assignee_id: { type: uuid, ref: user }
+      assignee_id: { type: uuid }
+    relations:
+      assignee: { type: belongsTo, table: user, foreignKey: assignee_id }
     render:
       template: default-list
       formatRow: '{{title}} → {{assignee.name}}'

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "latticesql",
-  "version": "3.4.6",
+  "version": "4.0.0",
   "description": "Persistent structured memory for AI agent systems — pluggable SQLite or Postgres backend, LLM context bridge",
   "type": "module",
   "main": "./dist/index.js",