latticesql 3.1.0 → 3.2.0

package/docs/architecture.md

@@ -0,0 +1,331 @@
+# Architecture Overview
+
+How `latticesql` is structured internally and the design decisions behind it.
+
+---
+
+## Table of Contents
+
+- [High-level picture](#high-level-picture)
+- [Module breakdown](#module-breakdown)
+- [Data flow](#data-flow)
+- [Design decisions](#design-decisions)
+- [Package structure](#package-structure)
+
+> **v0.5 additions** are called out inline below. They cover entity context directories, lifecycle management, the manifest, and the `reconcile()` method.
+
+---
+
+## High-level picture
+
+```
+                    ┌─────────────────────────────────────┐
+                    │              Lattice                 │
+                    │          (public facade)             │
+                    │                                      │
+                    │  defineEntityContext()  reconcile()  │  ← v0.5
+                    └──────────────────┬──────────────────┘
+                                       │
+          ┌────────────────────────────┼──────────────────────────┐
+          │                            │                          │
+┌─────────▼──────────┐   ┌────────────▼──────────┐  ┌───────────▼──────────┐
+│   SchemaManager     │   │    SQLiteAdapter       │  │   WritebackPipeline  │
+│                     │   │                        │  │                      │
+│ • define(table)     │   │ • open() / close()     │  │ • define(def)        │
+│ • defineEntityCtx() │   │ • run() / all() / get()│  │ • process()          │
+│ • getPrimaryKey()   │   │ • WAL mode             │  │ • file watching      │
+│ • getRelations()    │   │ • busy timeout         │  │ • dedup              │
+│ • applySchema()     │   └────────────────────────┘  └──────────────────────┘
+│ • applyMigrations() │
+└─────────────────────┘
+          │
+┌─────────▼──────────┐   ┌────────────────────────┐
+│   RenderEngine      │   │      SyncLoop           │
+│                     │   │                         │
+│ • render(outputDir) │◄──│ • watch(outputDir)      │
+│ • resolveRelations()│   │ • setInterval polling   │
+│ • writeFiles()      │   │ • cleanup on each tick  │  ← v0.5
+│ • _renderEntityCtxs │   │ • StopFn returned       │
+│ • writeManifest()   │   └─────────────────────────┘
+│ • cleanup()         │   ← v0.5
+└─────────────────────┘
+          │
+┌─────────▼──────────┐   ┌────────────────────────┐   ┌──────────────────────┐
+│   RenderTemplates   │   │      Sanitizer          │   │  Lifecycle (v0.5)    │
+│                     │   │                         │   │                      │
+│ • compileRender()   │   │ • sanitizeRow()         │   │ • readManifest()     │
+│ • default-list      │   │ • null-byte strip       │   │ • writeManifest()    │
+│ • default-table     │   │ • field length limits   │   │ • manifestPath()     │
+│ • default-detail    │   │ • audit event emission  │   │ • cleanupEntityCtxs()│
+│ • default-json      │   └─────────────────────────┘   └──────────────────────┘
+│ • interpolate()     │
+└─────────────────────┘
+          │
+┌─────────▼──────────┐
+│  EntityQuery (v0.5) │
+│                     │
+│ • resolveSource()   │
+│ • self / hasMany    │
+│ • manyToMany        │
+│ • belongsTo         │
+│ • custom            │
+│ • truncateContent() │
+└─────────────────────┘
+```
+
+---
+
+## Module breakdown
+
+### `Lattice` — public facade (`src/lattice.ts`)
+
+The single public class. It wires together all internal modules and exposes the public API. Key responsibilities:
+
+- Accepts `string | LatticeConfigInput` in the constructor, normalising both forms to a `dbPath` + table definitions
+- Enforces the `define → init → CRUD/sync` lifecycle (throws if called out of order)
+- Owns the event handler arrays and dispatches to them
+- Delegates every operation to an internal module
+
+### `SQLiteAdapter` — database layer (`src/db/sqlite.ts`)
+
+A thin wrapper around `better-sqlite3`. Responsibilities:
+
+- `open()` — opens the connection, optionally enabling WAL mode and setting busy timeout
+- `close()` — closes the connection
+- `run(sql, params)` — executes a DML statement (INSERT, UPDATE, DELETE)
+- `all(sql, params)` — returns `Row[]`
+- `get(sql, params)` — returns one `Row | undefined`
+- Exposes `.db` for the escape hatch
+
+The adapter is synchronous — `better-sqlite3` is a synchronous binding. All Lattice methods return Promises for a consistent async API surface, but they resolve synchronously.
+
+### `SchemaManager` — schema registry (`src/schema/manager.ts`)
+
+Holds all registered table and multi-table definitions. Responsibilities:
+
+- `define(table, compiledDef)` — stores a `CompiledTableDef` (render is always a function)
+- `defineMulti(name, def)` — stores a multi-table view definition
+- `getPrimaryKey(table)` — returns the PK column array for a table
+- `getRelations(table)` — returns the relations map for a table
+- `applySchema(adapter)` — emits `CREATE TABLE IF NOT EXISTS` for every registered table
+- `applyMigrations(adapter, migrations)` — creates `_lattice_migrations` and runs pending versions
+
+`CompiledTableDef` differs from `TableDefinition` in that the `render` field is always a compiled `(rows: Row[]) => string` function. The compilation happens once in `Lattice.define()` via `compileRender()`.
+
+### `RenderEngine` — sync to files (`src/render/engine.ts`)
+
+Executes the render cycle. Responsibilities:
+
+- `render(outputDir)` — iterates all registered tables, multi-table views, and entity context definitions; renders each to a string; writes to the appropriate output file (skipping unchanged content)
+- `resolveRelations(table, rows)` — for `belongsTo` relations referenced in template strings, joins to the related table in-process
+- `_renderEntityContexts(outputDir)` — (v0.5) renders all `defineEntityContext()` definitions; returns `Record<string, EntityContextManifestEntry>` and writes the manifest
+- `cleanup(outputDir, prevManifest, options, newManifest?)` — (v0.5) builds current slug sets from the DB and calls `cleanupEntityContexts()`
+- Returns `RenderResult` with file paths and timing
+
+File writes are skipped when the new content equals the existing file content — important for keeping LLM context file mtimes stable.
+
+### `EntityQuery` — entity source resolver (`src/render/entity-query.ts`) _(v0.5)_
+
+Contains the synchronous row-resolution logic for entity context directories. Responsibilities:
+
+- `resolveEntitySource(source, entityRow, entityPk, adapter)` — dispatches to the correct SQL query based on the source type (`self`, `hasMany`, `manyToMany`, `belongsTo`, `custom`)
+- `truncateContent(content, budget?)` — applies the per-file character budget and appends the truncation notice
+
+### `Lifecycle` — manifest and cleanup (`src/lifecycle/`) _(v0.5)_
+
+Two modules:
+
+- `manifest.ts` — `readManifest()`, `writeManifest()`, `manifestPath()` and the `LatticeManifest` / `EntityContextManifestEntry` types
+- `cleanup.ts` — `cleanupEntityContexts()` and the `CleanupOptions` / `CleanupResult` types
+
+The manifest (`{outputDir}/.lattice/manifest.json`) is the single source of truth for what Lattice generated. It is written atomically after every render cycle that includes entity contexts and read by the cleanup step to determine orphans.
+
+### `SyncLoop` — polling loop (`src/sync/loop.ts`)
+
+Wraps `RenderEngine` in a `setInterval` polling loop. Responsibilities:
+
+- `watch(outputDir, opts)` — starts the loop, returns a `StopFn`
+- Reads the previous manifest before each render cycle when `opts.cleanup` is set (v0.5)
+- Calls `engine.cleanup()` after each render cycle when `opts.cleanup` is set (v0.5)
+- Calls `onRender`, `onError`, and `onCleanup` callbacks per cycle
+
+### `WritebackPipeline` — agent-to-db writes (`src/writeback/pipeline.ts`)
+
+Monitors agent-written files and ingests new entries into the database. Responsibilities:
+
+- `define(def)` — registers a writeback definition
+- `process()` — reads registered files from their last-read offset, calls `parse()`, deduplicates, and calls `persist()` for each new entry
+
+### `RenderTemplates` — built-in templates (`src/render/templates.ts`)
+
+Contains the four built-in template implementations and the template compilation logic. Responsibilities:
+
+- `compileRender(def, table, schema, adapter)` — converts a `RenderSpec` into a `(rows: Row[]) => string` function. Called once at `define()` time
+- `interpolate(template, row, relations)` — replaces `{{field}}` and `{{rel.field}}` tokens
+- `renderList`, `renderTable`, `renderDetail`, `renderJson` — the four built-in renderers
+
+### `Sanitizer` — input safety (`src/security/sanitize.ts`)
+
+Applied to every row before it reaches the database. Responsibilities:
+
+- Strip null bytes from string values
+- Apply field length limits (`SecurityOptions.fieldLimits`)
+- Emit `AuditEvent` on each write operation
+
+### Config layer (`src/config/`)
+
+Two modules:
+
+- `types.ts` — TypeScript types for `LatticeConfig`, `LatticeEntityDef`, `LatticeFieldDef`, `LatticeEntityRenderSpec`
+- `parser.ts` — `parseConfigFile()` and `parseConfigString()`: read YAML, validate, convert to `ParsedConfig` (array of `{ name, TableDefinition }`)
+
+### Codegen layer (`src/codegen/`)
+
+- `generate.ts` — `generateTypes()`, `generateMigration()`, `generateAll()`: string generators for `types.ts` and `migration.sql`
+
+### CLI (`src/cli.ts`)
+
+Standalone entry point compiled to `dist/cli.js` with a `#!/usr/bin/env node` shebang. Uses no external CLI framework — just manual `process.argv` parsing. Calls `generateAll()` and logs results.
+
+---
+
+## Data flow
+
+### Insert flow
+
+```
+db.insert(table, row)
+  → Sanitizer.sanitizeRow(row)
+  → resolve PK (auto-UUID if default 'id' and absent)
+  → SQLiteAdapter.run(INSERT INTO ...)
+  → Sanitizer.emitAudit(table, 'insert', id)
+  → emit 'audit' event to handlers
+  → return Promise.resolve(pkValue)
+```
+
+### Render flow
+
+```
+db.render(outputDir) / SyncLoop (watch)
+  → RenderEngine.render(outputDir)
+    → for each registered table:
+        → SQLiteAdapter.all(SELECT * FROM table)
+        → apply table.filter (if defined)
+        → apply hooks.beforeRender (if defined)
+        → resolve belongsTo relations (for template interpolation)
+        → call compiled render function(rows)
+        → compare output to existing file content
+        → write file if changed
+    → for each multi-table view:
+        → call def.keys() for anchor rows
+        → for each anchor: query tables, call def.render(key, tables)
+        → write files
+    → _renderEntityContexts(outputDir):        ← v0.5
+        → for each entity context definition:
+            → render index file (if defined)
+            → for each entity row:
+                → derive slug → entity subdirectory
+                → for each EntityFileSpec:
+                    → resolveEntitySource(source, row, pk, adapter)
+                    → call spec.render(rows)
+                    → apply budget truncation (if defined)
+                    → skip write if omitIfEmpty and rows empty
+                    → write file if content changed
+                → write combined file (if defined)
+        → writeManifest(outputDir, manifest)
+  → return RenderResult
+```
+
+### Reconcile flow _(v0.5)_
+
+```
+db.reconcile(outputDir, options)
+  → prevManifest = readManifest(outputDir)   ← read BEFORE render
+  → RenderEngine.render(outputDir)           ← writes new manifest
+  → newManifest = readManifest(outputDir)
+  → cleanupEntityContexts(
+      outputDir,
+      entityContexts,
+      currentSlugsByTable,       ← fresh DB query
+      prevManifest,
+      options,
+      newManifest                ← used to detect omitIfEmpty-skipped files
+    )
+  → return ReconcileResult { ...renderResult, cleanup: CleanupResult }
+```
+
+### Sync flow
+
+```
+db.sync(outputDir)
+  → RenderEngine.render(outputDir)     ← same as render()
+  → WritebackPipeline.process()        ← read agent files, ingest entries
+  → return SyncResult
+```
+
+---
+
+## Design decisions
+
+**Synchronous SQLite, async API surface.** `better-sqlite3` is synchronous. All Lattice methods still return `Promise<T>` — this allows callers to use `await` and keeps the API contract stable if an async adapter is ever added. The promises resolve in the same tick.
+
+**Compile at define-time, not render-time.** `compileRender()` converts a `RenderSpec` (which can be a string, object, or function) into a single `(rows: Row[]) => string` function when `define()` is called. This ensures zero per-cycle overhead for template dispatch.
+
+**No ORM, no query builder.** Lattice does not attempt to abstract SQL. The `columns` spec is a raw SQLite type string. Advanced queries use `db.db` (escape hatch). This keeps the library small and avoids the impedance mismatch that plagues ORMs.
+
+**Config form is a thin wrapper over `define()`.** `new Lattice({ config })` calls `parseConfigFile()` then loops over the result calling `this.define()`. The config form is not a separate code path — it just automates the manual `define()` calls.
+
+**Files are skipped when content is unchanged.** `RenderEngine` compares the rendered string to the file's current content before writing. This prevents unnecessary filesystem writes and keeps file modification times stable (important for LLM context systems that watch mtimes).
+
+**Relation resolution happens in-process.** When a `belongsTo` relation is referenced in a `{{rel.field}}` token, Lattice issues a `SELECT` for each row via the adapter. This is intentionally simple — N+1 for N rows. For tables with thousands of rows this could be slow, but Lattice is designed for small-to-medium context tables (dozens to hundreds of rows), not analytics workloads.
+
+**Manifest-driven cleanup.** Lattice never scans the output directory for files it does not recognise. Instead it only removes files and directories it previously recorded in the manifest. This means files created by agents or other tools in entity directories are never touched — unless they happen to match a filename Lattice manages, which is why `protectedFiles` exists as an escape valve.
+
+**Render before cleanup.** `reconcile()` always runs the render cycle first, writing a new manifest, before running cleanup. This ensures the cleanup step has both the previous state (what to remove) and the current state (what is still being written) and can correctly detect `omitIfEmpty` files that were skipped this cycle but existed before.
+
+---
+
+## Package structure
+
+```
+src/
+├── index.ts              # Public exports
+├── lattice.ts            # Lattice class (public facade)
+├── types.ts              # All public TypeScript types
+├── cli.ts                # CLI entry point
+├── config/
+│   ├── types.ts          # YAML config schema types
+│   └── parser.ts         # parseConfigFile / parseConfigString
+├── codegen/
+│   └── generate.ts       # generateTypes / generateMigration / generateAll
+├── db/
+│   └── sqlite.ts         # SQLiteAdapter
+├── schema/
+│   └── manager.ts        # SchemaManager
+├── render/
+│   ├── engine.ts         # RenderEngine (+ entity context rendering, v0.5)
+│   ├── templates.ts      # Built-in templates + compileRender + interpolate
+│   └── entity-query.ts   # resolveEntitySource + truncateContent (v0.5)
+├── lifecycle/             # v0.5
+│   ├── index.ts          # Barrel export
+│   ├── manifest.ts       # readManifest / writeManifest / manifestPath
+│   └── cleanup.ts        # cleanupEntityContexts + CleanupOptions/Result
+├── sync/
+│   └── loop.ts           # SyncLoop (+ cleanup integration, v0.5)
+├── writeback/
+│   └── pipeline.ts       # WritebackPipeline
+└── security/
+    └── sanitize.ts       # Sanitizer
+
+tests/
+├── unit/
+│   ├── config.test.ts        # parseConfigFile / parseConfigString
+│   ├── codegen.test.ts       # generateTypes / generateMigration + integration
+│   ├── lattice.test.ts       # Core CRUD / query / render tests
+│   └── entity-query.test.ts  # resolveEntitySource unit tests (v0.5)
+├── integration/
+│   ├── entity-context.test.ts # defineEntityContext() flow (v0.5)
+│   └── lifecycle.test.ts      # reconcile() + cleanup (v0.5)
+└── fixtures/
+    └── lattice.config.yml
+```

package/docs/assistant.md

@@ -0,0 +1,138 @@
+# The AI assistant & Context Constructor (2.0+)
+
+`lattice gui` ships an optional assistant rail. It is **GUI-only** and inert
+until you configure a credential — the `latticesql` library API is unchanged.
+
+## Connect Claude
+
+Open **Settings → User → Assistant** and paste an Anthropic API key, or set
+`ANTHROPIC_API_KEY` in the environment. Keys are stored encrypted in the native
+`secrets` entity; the env var is a fallback. That's all the chat and the
+Context Constructor need.
+
+A **"Connect your Claude subscription"** link (Authorization-Code + PKCE) appears
+only when all four `ANTHROPIC_OAUTH_*` values are configured (see
+[`.env.example`](../.env.example)); otherwise the panel shows a dormant hint.
+Use a fixed GUI port so the redirect URI is stable: `lattice gui --port 4317`.
+
+## Chat
+
+The rail runs a Claude tool-calling loop streamed over SSE. The model can list,
+read, **full-text search**, create, update, link, delete tables, and revert in
+the active database. **Every edit goes through the same audited, undoable
+mutation path as a manual edit** — it appears in the activity feed and the
+version history and can be reverted.
+
+The **top search box hands your query to the assistant**: type and press Enter
+and the query is submitted as a chat turn, which the assistant answers using its
+`search` (and read) tools rather than a plain text match. The assistant never
+sees the conversation-storage or `secrets` tables (search and `list_entities`
+both exclude them).
+
+When the assistant points you at a specific record — ask it to "link me to" or
+"open" one — it renders a **clickable object pill** inline in its answer
+(emitted as `[label](lattice://<table>/<id>)`). Clicking the pill opens that row
+in the GUI via the same mode-aware navigator the activity feed uses; it links the
+user-facing record (the contract/person/etc.) rather than an internal `files` id,
+and only ids it actually retrieved.
+
+**Deleting a table is guarded + reversible.** The `delete_entity` tool refuses
+built-in tables, tables another table links to, and tables you don't own. An
+**empty** table is soft-deleted immediately; a **non-empty** one is **not**
+deleted until you decide what happens to the data — the tool reports the row
+count and the assistant asks, then you choose `delete_data` (soft-delete the rows
+too) or `move_to` another table. The physical table + rows are kept (no hard
+drop), so the whole thing is revertible from version history.
+
+Conversations persist in the native `chat_threads` / `chat_messages` entities;
+use the thread switcher to revisit them. A new thread is **named from a short AI
+summary** of its first exchange (e.g. "Adding New Notes About Cheese"). The
+assistant's **data changes are saved with each turn and replayed as activity
+cards** when you reopen the conversation — collapsed by type (e.g. "Deleted 19
+tables", "Removed 49 rows across 9 tables"), with the operation's icon. Reads
+(list / get / search) change nothing, so they produce no card; only data changes
+appear. The activity feed is scoped to the open conversation rather than a global
+workspace log.
+
+The assistant **remembers what it read across turns.** Earlier tool calls and
+their results (including row ids) are replayed into the model's context, so a
+follow-up like "now update that row" reuses the id it just listed instead of
+guessing one. Replay is bounded to the recent turns within a size budget and is
+secret-redacted; set `LATTICE_CHAT_REHYDRATE=false` to disable it. Reads are also
+deterministically ordered, so listing the same table twice returns the same rows.
+
+The assistant **knows the record you're looking at.** When a file or row detail is
+open, the chat passes that record (table + id) as context, so "delete this file",
+"summarize this", or "share this row" act on it directly instead of asking which
+one. It's a hint only — every action still goes through the same permission-gated
+tools, so it can't reach a record you couldn't otherwise touch.
+
+The assistant can also **answer questions about Lattice itself.** Ask "what is
+private mode?" or "how do I invite a member?" and it calls the `lattice_help` tool,
+which searches Lattice's own documentation (these `docs/*.md` files — the single
+canonical source, shipped in the npm package) and answers from it rather than
+guessing or searching your data.
+
+## The Context Constructor (file & text ingest)
+
+Drag files onto the rail, click the paperclip, or paste text (or a URL). For each
+source:
+
+1. **Referenced, not copied.** The source becomes a native `files` row that
+   points at the original; bytes are not moved into Lattice.
+2. **Extracted.** Plain text/markdown/code is read directly; documents
+   (PDF, Word `.docx`/`.doc`, PowerPoint `.pptx`, Excel `.xlsx`, OpenDocument
+   `.odt`/`.ods`/`.odp`, EPUB, RTF) are parsed **natively in-process** — no
+   external CLI; **images are described by Claude vision**; **scanned/image-only
+   PDFs** with no text layer fall back to Claude's native PDF read; a pasted
+   **bare URL is crawled** for readable text (and the URL preserved on the row as
+   a `cloud_ref`). Legacy binary `.xls`/`.ppt` (pre-2007) and any other binary
+   are still referenced and marked `extraction_status='skipped'`. The parsers
+   ship as optional dependencies, so a document just skips (rather than failing)
+   if its parser isn't installed.
+3. **Summarized** with Claude Haiku (the description fills in).
+4. **Organized.** The text is classified against your existing records, and for
+   each match the file is **linked** — **auto-creating the `files_<entity>` junction
+   table when none exists yet**. When a source fits **nothing** (and aggressiveness
+   is high), a new native `notes` object is **created** for it, linked back via
+   `source_file_id`. New objects, enrichment, links, and junctions are all
+   reversible via the version history.
+
+### Library API
+
+The same intelligence is a first-class, GUI-independent API (inert without an LLM
+client): `organizeSource`, `describeImage`, `crawlUrl`, `enrichKnowledge`, and the
+`summarizeText` / `classifyLinks` primitives — all importable from `latticesql`.
+`sharp` + `file-type` are optional, lazily-loaded deps; the crawler uses `jsdom` +
+`@mozilla/readability`.
+
+A transient **"Analyzing…"** row shows while ingest runs; the add/enrich/link
+events stream into the feed as the server materializes them.
+
+## Inference Aggressiveness
+
+A single **Conservative ↔ Aggressive** slider (Settings → Assistant) tunes how
+much the assistant extrapolates. It maps to the model sampling temperature, how
+liberally the ingest classifier proposes links, and whether ingest auto-creates a
+missing junction (gated at ≥ 0.25) versus only suggesting it. Default: balanced
+(0.5). Settable via `PUT /api/assistant/aggressiveness { "value": 0..1 }`. This
+is a **user preference** (machine-local `~/.lattice/preferences.json`), not a
+workspace secret — it persists across workspaces and never appears in a
+workspace's Secrets object.
+
+## Voice (optional)
+
+Set `OPENAI_API_KEY` (Whisper) or `ELEVENLABS_API_KEY` to enable the composer
+mic; choose the provider in the Assistant settings (also a machine-local user
+preference, not a workspace secret). When no microphone is available the mic
+button is shown disabled with a tooltip rather than erroring. **While a note is
+recording or transcribing, the composer is read-only** — it shows a
+"Listening… / Transcribing…" placeholder and the Send button is disabled — and
+the transcript is inserted when you stop.
+
+## Cloud
+
+The assistant runs against local SQLite and any `postgres://` connection, including
+a Lattice cloud. On a cloud it connects as your own scoped role, so its reads and
+writes are confined by Postgres Row-Level Security to the rows you may see — see
+[cloud.md](cloud.md).