latticesql 4.3.7 → 5.0.0

package/docs/computed-tables.md

@@ -0,0 +1,200 @@
+# Computed tables
+
+A **computed table** is a live, read-only view built from the tables you already
+have. You describe _what each field is_ — a copied column, a calculation, an
+AI-derived value, or a total across linked rows — and Lattice compiles that
+description into a SQL VIEW and registers it as a queryable table. The values
+are never copied: every read reflects the current state of the records the view
+is built from.
+
+Everything here is **additive and opt-in**: a workspace with no `computed:`
+section behaves exactly as before.
+
+## The shape of a definition
+
+A computed table has one **base** table (a declared entity or another computed
+table) and a set of named **fields**. The view always projects the base's
+primary key as `id` first, then each field in declaration order:
+
+```yaml
+computed:
+  ticket_summary:
+    base: tickets
+    fields:
+      title: { kind: alias, source: title }
+      team: { kind: alias, source: assignee.team.name }
+      is_urgent: { kind: calc, expr: 'priority >= 3', type: boolean }
+      tag_count: { kind: aggregate, via: ticket_tags.tag, fn: count }
+```
+
+Definitions live in the workspace config next to `entities:`. The GUI builder
+(below) reads and writes the same section — a hand-edited config and a
+GUI-built view are the same thing.
+
+## The five field kinds
+
+### `alias` — copy a field
+
+Project a column of the base table, or follow declared `belongsTo` relations
+with a dotted path:
+
+```yaml
+title: { kind: alias, source: title }
+team: { kind: alias, source: assignee.team.name } # tickets → people → teams
+```
+
+### `calc` — a calculation
+
+A sandboxed expression over base columns and dotted paths. Expressions support
+arithmetic, comparisons, `and`/`or`/`not`, `case … when`, `cast`, and a fixed
+function set: `coalesce`, `nullif`, `lower`, `upper`, `trim`, `length`,
+`substr`, `replace`, `abs`, `round`. Raw SQL never passes through — the
+expression is parsed and re-emitted, so anything outside the grammar is
+rejected at definition time.
+
+```yaml
+is_urgent: { kind: calc, expr: 'priority >= 3', type: boolean }
+label: { kind: calc, expr: "coalesce(nickname, name, 'unknown')" }
+```
+
+`type` is the display type (defaults to `text`).
+
+### `ai_classify` — an AI-assigned category
+
+A model assigns each row one label from a fixed set, based on one input field:
+
+```yaml
+sentiment:
+  kind: ai_classify
+  input: body
+  prompt: How does the customer feel in this message?
+  labels: [happy, neutral, frustrated]
+```
+
+Each **distinct input value** is labeled once and the result is materialized
+into a bookkeeping table the view joins against — the model is never re-run at
+read time, and two rows with the same input always get the same label.
+
+### `ai_transform` — AI-written text
+
+A model derives a free-form value from one or more input fields (input order is
+part of the identity — reordering inputs is a new definition):
+
+```yaml
+summary:
+  kind: ai_transform
+  inputs: [title, body]
+  prompt: Summarize this ticket in one sentence.
+```
+
+Results are cached per row, keyed on the current input values — see
+[staleness](#refreshing-ai-values--staleness) below.
+
+### `aggregate` — a total across links
+
+Fold many linked rows into one value per base row, through a junction table:
+
+```yaml
+tag_count: { kind: aggregate, via: ticket_tags.tag, fn: count }
+total_paid: { kind: aggregate, via: invoice_payments.payment, fn: sum, column: amount }
+```
+
+`via` is `<junctionTable>.<remoteRelation>`; `fn` is one of `count`, `sum`,
+`avg`, `min`, `max`, `concat`; `column` names the remote column to aggregate
+(required for every `fn` except `count`).
+
+## Building one in the GUI
+
+Open the **Tables** tab. The explorer's third column is **Computed Tables**;
+its header carries a **+ New** button that opens the builder (`#/computed/new`).
+
+1. **Name it** — lowercase letters, numbers, and underscores.
+2. **Pick the base table** ("Built from"). Fields can come from the base and
+   anything linked to it — the pickers list every reachable column, including
+   dotted paths through linked tables and aggregate targets through junctions.
+3. **Add fields.** Each row is a name plus a kind — _Copy a field_,
+   _Calculation_, _AI category_, _AI text_, or _Total across links_ — with the
+   inputs that kind needs.
+4. **Preview.** A dry run compiles the definition against your live schema and
+   shows up to 20 sample rows. Nothing is created or saved by a preview. Each
+   field is stamped ✓ (compiled) or ✕ (the failing field), and the compiled
+   SELECT is available under the collapsed **Definition (SQL)** block.
+5. **Create.** Enabled once the current definition has previewed successfully
+   (any edit re-requires a preview). Creating registers the view, persists the
+   definition to the workspace config, and lands you on the view's rows.
+
+Editing an existing view (**Edit definition →** in a computed card's detail
+panel, or `#/computed/<name>`) is the same form plus two more actions:
+**Refresh values** (runs the AI fill now, streaming per-field progress) and
+**Remove**.
+
+The explorer also shows the projection itself: a dashed connector from the base
+table to the computed view, and the view's detail panel lists the base under
+_Upstream · sources_.
+
+## Refreshing AI values + staleness
+
+AI fields are **materialized, never computed at read time**. A background fill
+runs when a definition with AI fields is created or changed; you can also run
+it on demand (**Refresh values** in the builder, or **Refresh** in the Tables
+explorer's detail panel).
+
+Staleness is handled by construction: an AI value is cached against the exact
+input values it was derived from. When a source row changes, its cached value
+no longer matches, so the field reads **blank for that row until the next
+refresh** — you may see a gap, but never a stale value. Unchanged rows keep
+their cached values (no re-billing for what's already known), and a changed
+prompt, label set, input list, or model invalidates exactly that field's cache.
+
+## Read-only rows
+
+A computed table's rows are derived, not authored. The GUI marks the view with
+a **Computed** badge on its rows page and record pages, shows where the values
+come from, and offers no editing affordances (no field editor, no markdown
+editing, no per-row delete). The server enforces the same rule: any direct
+write to a computed table is refused with a message pointing you at the source
+tables or the definition. To change what a computed table shows, change the
+records it is built from — or edit its definition.
+
+The source tables are protected in the other direction too: deleting or
+renaming a table is refused while a computed table reads from it, naming the
+definitions that depend on it.
+
+## Deleting + undo
+
+Removing a computed view drops the view and its definition (and its AI-value
+cache). Nothing about the source tables changes. The operation is recorded in
+version history and is **revertible** — undo re-creates the view from the
+captured definition. A computed table that other computed tables are built on
+cannot be deleted until they are deleted or repointed.
+
+A refresh, by contrast, only fills in AI-derived cells; it appears in history
+as an informational entry with nothing to revert.
+
+## HTTP API
+
+The GUI drives computed tables through `lattice gui`'s local server. On a team
+cloud, mutating verbs (and preview/refresh, which belong to the owner-side
+builder) are owner-only — a scoped member gets a 403.
+
+| Method | Endpoint                             | What it does                                                            |
+| ------ | ------------------------------------ | ----------------------------------------------------------------------- |
+| GET    | `/api/computed-tables`               | List definitions with per-field fill/error state                        |
+| GET    | `/api/computed-tables/:name`         | One definition plus its compiled SELECT (`{ def, sql }`)                |
+| GET    | `/api/computed-tables/fields?base=`  | Field-picker candidates for a base: columns, dotted paths, aggregates   |
+| POST   | `/api/computed-tables/preview`       | Dry-run a definition (`{ def, limit? }`) — no DDL, nothing persisted    |
+| POST   | `/api/computed-tables`               | Create (`{ name, def }`) — registers the view + persists the definition |
+| PUT    | `/api/computed-tables/:name`         | Update the definition (`{ def }`); dependents recompile in order        |
+| DELETE | `/api/computed-tables/:name`         | Delete the view + definition (refused while dependents exist)           |
+| POST   | `/api/computed-tables/:name/refresh` | Run the AI fill now; streams per-field progress as NDJSON               |
+
+The preview response is `{ columns, rows, sql, fieldTypes, pendingAi }` —
+`pendingAi` maps each AI field to how many items a fill pass would enqueue.
+The refresh stream emits one JSON object per line:
+`{ phase: 'field', field, message }` when a field starts,
+`{ phase: 'field-done', field, filled, pending, error? }` when it finishes,
+and a final `{ done: true }` (or `{ phase: 'error', message }`).
+
+Definition shape errors, unknown bases/columns, and dependency refusals come
+back as `400 { error }` with the compiler's message verbatim — the same text
+the builder shows in its error strip.

package/docs/connectors.md

@@ -1,166 +1,162 @@
-# Connectors
+# MCP Connectors
 
 Connectors sync data from external systems into Lattice as **connected data
 types** — tables whose rows are ingested from a source rather than authored
-locally. They make external data first-class Lattice data: queryable, full-text
-searchable, rendered to context, linked on the graph, and ACL-scoped on a cloud.
-
-A _connector_ talks to one external product (a _toolkit_) and exposes its object
-types as connected data types. The built-in connector is **Jira**, which talks to
-Jira Cloud's REST + Agile APIs directly via [`jira.js`](https://github.com/MrRefactoring/jira.js)
-using your own Atlassian credentials — there is no broker service and no extra
-API key. Adding another source (Gmail, Slack, …) is a new `Connector`
-implementation, not changes to the core.
-
-> `jira.js` is an **optional dependency**. The package compiles and runs without
-> it; it is loaded lazily and a clear error is thrown only when the Jira connector
-> is actually used. Install it to use the connector: `npm install jira.js`.
-
-## Concepts
-
-| Concept                 | Meaning                                                                                                                |
-| ----------------------- | ---------------------------------------------------------------------------------------------------------------------- |
-| **Connector**           | An implementation of the fetch/auth SPI (e.g. the Jira connector).                                                     |
-| **Toolkit**             | An external product a connector serves (e.g. `jira`).                                                                  |
-| **Connected data type** | A Lattice table with a `source` descriptor — its rows are synced from a toolkit model.                                 |
-| **Connector instance**  | A registered connection (`__lattice_connectors` row): which toolkit, the per-member connection handle, and sync state. |
-
-A connected table's **natural key is its primary key** (a stable external id), so
-re-syncs upsert idempotently. Every row carries connector lineage:
-`_source_connector_id` and `_source_model` (immutable) plus `_source_synced_at`.
-
-## Credentials
-
-The Jira connector authenticates as **you**, with an Atlassian **API token**
-(HTTP Basic auth: your account email + the token). Create one at
-<https://id.atlassian.com/manage-profile/security/api-tokens>. Lattice validates
-the credentials against Jira on connect (`GET /myself`) and stores them in the
-**machine-local encrypted credential store** — they never leave the machine
-except to call the Jira API directly, and they are never written to the registry,
-logs, or any rendered context.
-
-## Using the GUI
-
-Open **Settings → Connectors**, fill in your Jira **site URL**
-(`https://your-domain.atlassian.net`), **account email**, and **API token**, then
-**Connect**. The credentials are validated and the initial sync runs. Each
-connected toolkit shows its status + last-synced time with **Refresh** and
-**Disconnect** buttons, and connected data types get a "Connected" badge in the
-Objects list.
-
-## Quick start (programmatic)
+locally. Every connector is an **MCP connection**: Lattice runs as a local
+[Model Context Protocol](https://modelcontextprotocol.io) client and talks to
+the server directly from your machine. There is no provider-specific connector
+code and no broker — a provider is just another MCP server URL.
+
+## Connecting a server (GUI)
+
+Open **Configure → MCP Connectors**. Every connection you have lives in this
+tab: name (the server's self-reported name from the MCP handshake), URL, status,
+last sync, and per-server **Refresh** / **Disconnect** / **Reconnect** actions.
+
+To add one, paste the server's URL into **Add an MCP connector** and click
+**Connect**:
+
+- **Open servers** (no auth) connect and sync immediately.
+- **OAuth servers** open the provider's own sign-in in your browser. Approve it
+  and return to Lattice — the connection completes on a loopback callback and
+  the first sync runs.
+- **Servers that require a pre-registered OAuth client** (no dynamic
+  registration and no client-ID-metadata-document support) get a clear prompt:
+  the form reveals **client ID / client secret** fields for the credentials the
+  provider issued you, then the same OAuth sign-in runs with them.
+
+Each added server is its own connection — connect as many as you like,
+side by side. **Disconnect** soft-deletes the synced rows (recoverable), prunes
+their rendered context, revokes the stored tokens, and keeps the server URL so
+**Reconnect** can re-run the sign-in without re-entering anything.
+
+## How OAuth client identity works
+
+Lattice identifies itself to a provider's authorization server using the first
+mechanism the server supports, in this order:
+
+1. **A stored client** — either from a previous dynamic registration or the
+   client ID you entered by hand.
+2. **Client-ID metadata document (CIMD)** — the modern MCP mechanism: the
+   client_id IS a stable HTTPS URL pointing at a static JSON document that
+   describes the app (name, redirect URIs, grant types). Lattice's document is
+   hosted at `https://latticedesktop.com/oauth/client-metadata.json`.
+3. **Dynamic client registration** (RFC 7591), for servers that offer a
+   registration endpoint.
+
+Every flow is a PKCE (S256) authorization-code flow for a public client; tokens
+are refreshed automatically.
+
+### The privacy model
+
+The hosted metadata document is **static app identity, not a data plane**:
+
+- It is the same fixed file for every Lattice install, contains **zero user
+  data**, and is never written.
+- Only the **provider's authorization server** fetches it (to answer "who is
+  this client asking for access?"). Your machine and your browser never touch
+  the hosting domain during the flow.
+- MCP traffic flows **directly** between your local Lattice and the MCP server —
+  nothing is proxied through any Lattice-hosted service.
+- OAuth tokens live only in the machine-local encrypted credential store
+  (AES-256-GCM under a machine-local master key). They never enter the
+  workspace database, API responses, or logs.
+
+Self-hosters can point `LATTICE_MCP_CLIENT_METADATA_URL` at their own document;
+setting it to an empty string disables the mechanism entirely (dynamic
+registration and manual client entry still work).
+
+## What gets synced
+
+The connector introspects the server and pulls its readable content into one
+connected table, **`mcp_items`**:
+
+| column    | meaning                                                               |
+| --------- | --------------------------------------------------------------------- |
+| `item_id` | natural key (`<tool>:<id>` for items, `resource:<uri>` for resources) |
+| `kind`    | `item` (from a read tool) or `resource` (from `resources/list`)       |
+| `tool`    | the MCP tool that produced an item                                    |
+| `server`  | the server's hostname — items from different servers stay apart       |
+| `title`   | best-effort title                                                     |
+| `summary` | best-effort description/snippet                                       |
+| `data`    | the raw JSON payload (for resources: `{ uri, mimeType }`)             |
+
+Discovery calls the server's `tools/list`, invokes each **no-argument read
+tool** (bounded per tool; obviously write-shaped tools are never called), and
+then lists the server's advertised **resources** — its "available files" — via
+the standard `resources/list`. Rows are full Lattice rows: queryable, full-text
+searchable, rendered to context, per-member `private` by default, and stamped
+with immutable lineage (`_source_connector_id`, `_source_model`). A re-sync
+upserts on the natural key and soft-deletes rows that vanished from the source.
+
+Freshness is pull-based — there is no background scheduler. Syncs run on
+connect, on **Refresh**, and on GUI load when the last sync is older than an
+hour.
+
+## Library API
+
+The same engine is a library surface:
 
 ```typescript
 import {
-  JiraConnector,
+  genericConnector,
   createConnector,
   syncConnector,
   syncIfStale,
   disconnectConnector,
 } from 'latticesql';
 
-const connector = new JiraConnector();
-
-// Validate + store the member's Atlassian credentials, returning a connection handle:
-const { connectionId, displayName } = await connector.connect({
-  site: 'https://your-domain.atlassian.net',
-  email: 'you@example.com',
-  apiToken: process.env.JIRA_API_TOKEN!,
-});
+const connector = genericConnector();
 
+// GUI-less flows drive beginConnect/completeConnect (OAuth) or connect a local
+// stdio server; the registry + sync engine are shared with the GUI routes:
 const connectorId = await createConnector(db, {
-  connector: 'jira',
-  toolkit: 'jira',
-  displayName: displayName ?? 'jira',
+  connector: 'mcp',
+  toolkit: 'mcp',
+  displayName: 'My MCP server',
   connectionRef: connectionId,
   connectedBy: 'user-123',
 });
-
-await syncConnector(db, connector, connectorId); // defines the tables + ingests
-```
-
-## Sync model
-
-- **`syncConnector(db, connector, connectorId)`** — full sync: paginated +
-  bounded fetch per model, idempotent upsert with lineage stamping, soft-delete
-  of rows that vanished from the source, and graph-edge derivation from FK
-  columns. Records the outcome on the connector; an external-sync failure is
-  re-thrown (never swallowed).
-- **`syncIfStale(db, connector, connectorId, maxAgeMs?)`** — no-op if the last
-  sync is within `maxAgeMs` (default 1 hour). Call it on app load.
-- **`syncStaleConnectors(db, connector, maxAgeMs?)`** — sync every stale
-  connector this implementation serves. The GUI calls this on load, so connected
-  data refreshes hourly **without a scheduler**. Manual refresh forces a sync.
-
-Reads done by the sync engine are bounded and column-projected — it never scans a
-full table to diff. Per-parent models iterate the parent's already-synced keys
-(comments are fetched per issue; sprints per board).
-
-## Disconnecting
-
-```typescript
-await disconnectConnector(db, connector, connectorId, { outputDir });
+await syncConnector(db, connector, connectorId);
+await syncIfStale(db, connector, connectorId); // re-sync when stale (1h)
+await disconnectConnector(db, connector, connectorId); // soft teardown
 ```
 
-Soft-deletes every ingested row (children before parents), prunes rendered
-context files, marks the connector `disconnected` (use `{ mode: 'hard' }` to also
-remove the registry row), and drops the stored credentials. Soft-deleted rows
-drop out of the rendered context, full-text search, and the GUI listings (all of
-which filter `deleted_at IS NULL`), and their graph edges are removed — so the
-data is no longer available to the agent while remaining physically present and
-restorable.
-
-## Cloud ACL
-
-On a cloud (Postgres) workspace, the **owner** scopes connected data per member:
-
-```typescript
-await enableConnectorRls(db, connector, 'jira');
-```
-
-This enables Row-Level Security on the registry and the toolkit's connected
-tables and applies each type's default visibility — `private` (visible only to
-the connecting member) or `everyone` (shared with the team). It is a no-op on
-SQLite, a non-cloud Postgres, or for a non-owner role.
-
-Because each member connects with their own Atlassian credentials, the data a
-member syncs is already scoped to what they can see in the source — per-user
-auth, not a shared admin credential. Derived enrichment over connected rows
-inherits the source's visibility automatically: write it through
-`db.observe(table, pk, { changeKind: 'derived', sourceRef: [connectedRowId] })`,
-and the source-gated fold hides it from a viewer who can't see the source.
-
-## The Jira toolkit
-
-Connecting Jira creates six connected data types:
-
-| Table           | Natural key | Notes                                                         |
-| --------------- | ----------- | ------------------------------------------------------------- |
-| `jira_projects` | project key | FTS on name/description                                       |
-| `jira_issues`   | issue key   | FTS on summary/description; edges → project, assignee, sprint |
-| `jira_comments` | comment id  | fetched per issue; edges → issue, author                      |
-| `jira_users`    | account id  |                                                               |
-| `jira_boards`   | board id    | edge → project                                                |
-| `jira_sprints`  | sprint id   | fetched per board; edge → board                               |
-
-## Adding a connector
-
-Implement the `Connector` SPI and point the GUI/registry at it. The SPI is small:
-
-- `connector` / `toolkits()` / `models(toolkit)` — identity + the connected
-  `ConnectedModelDef`s (table schema, natural key, FK relations → graph edges).
-- `listChanges(toolkit, model, ctx)` — an async iterable of normalized
-  `ExternalRecord`s for one model, **paged and bounded** (per-parent models read
-  `ctx.parentKey`).
-- `disconnect(connectionRef)` — revoke / drop the stored connection.
-- `authorize` / `completeAuth` — for an OAuth-redirect source. A
-  credential-based connector (like Jira) instead exposes a `connect(creds)`
-  method that validates + stores the credentials and returns a connection handle;
-  the GUI route calls it when the connector supports it.
-
-The sync engine, graph wiring, teardown, and ACL all work from the model
-descriptors — no further code is required.
-
-```
-
-```
+Library consumers embedding a specific provider can use
+`introspectiveConnector(spec)` (the same engine pre-pointed at a fixed endpoint
+with its own table name) or `SimpleMcpConnector` (hand-typed models + per-model
+tool bindings) — see `src/connectors/types.ts` for the SPI.
+
+On a cloud workspace, the owner's `enableConnectorRls(db, connector, 'mcp')`
+scopes connected rows per member (private by default).
+
+> `@modelcontextprotocol/sdk` is an **optional dependency**. The package
+> compiles and runs without it; it is loaded lazily and a clear error is thrown
+> only when an MCP connector is actually used. Install it to use connectors:
+> `npm install @modelcontextprotocol/sdk`.
+
+## Security
+
+- **SSRF guard**: every HTTP hop — the server URL, redirects, and each
+  OAuth-discovered endpoint — is DNS-resolved and re-validated against a
+  private/loopback/link-local/metadata-address blocklist before it is fetched.
+- **Token storage**: machine-local encrypted file, never the shared DB.
+- **Error hygiene**: sync errors surfaced in the GUI are sanitized so a raw DB
+  constraint error can never echo another member's data.
+
+## Troubleshooting
+
+- **"Incompatible auth server: does not support dynamic client registration"** —
+  the server supports neither CIMD nor dynamic registration. The GUI's add form
+  reveals client ID/secret fields; create an OAuth client in the provider's
+  admin console and paste its credentials.
+- **The browser shows a provider error at sign-in** — the provider's
+  authorization server could not validate the client. If you overrode
+  `LATTICE_MCP_CLIENT_METADATA_URL`, make sure that URL is publicly reachable
+  and serves JSON.
+- **"Sign-in didn't complete"** — the loopback callback never arrived (browser
+  closed mid-flow, or the sign-in was abandoned). Click Connect again.
+- **A server connects but syncs nothing** — it may expose only tools that
+  require arguments (skipped in introspective mode) and no resources. The
+  connection is still valid; data arrives when the server offers no-argument
+  read tools or resources.

package/docs/desktop.md

@@ -38,6 +38,15 @@ browser tab.
   while keeping the auth session local to the app.
 - **Upgrade-on-run.** On launch the app checks the release channel via
   `Deno.autoUpdate()` and the `latest.json` manifest published with each release.
+- **Update-available hint while running.** A window left open for a long time would
+  otherwise miss releases that ship after launch, so the GUI also polls the same
+  `latest.json` (read-only — no download or relaunch) and shows an **"Update
+  available — Restart to update"** link next to the version chip when a newer
+  release is published. Clicking it runs the bundled updater (download + relaunch).
+- **Disable auto-update.** Set `LATTICE_NO_AUTO_UPDATE=1` to pin the app to its
+  current version — no manifest probe, no `Deno.autoUpdate()`, no relaunch (the
+  desktop equivalent of the CLI's `lattice gui --no-auto-update`). Useful for
+  testing, air-gapped, or reproducible-demo runs.
 - **Upgrade-on-install.** The download page always links the latest release, so a
   fresh install is always current.
 
@@ -66,6 +75,45 @@ Releases are cut by the `Desktop Release` workflow on a `v*` tag: it builds both
 OSes, generates `latest.json`, and uploads the installers + manifest to the
 GitHub Release.
 
+### Code signing (maintainers)
+
+Builds are **ad-hoc (unsigned) by default** — contributors need no certificates,
+and the unsigned `.pkg` still installs to `/Applications` (with the macOS
+"unidentified developer" prompt on first launch). Real signing activates only
+when credentials are supplied through the environment; no identity, team, or key
+value is hardcoded anywhere in the repo.
+
+To produce a signed + notarized macOS build locally, export before running
+`npm run desktop:build:mac:pkg`:
+
+- `SIGN_APP_IDENTITY` — the Developer ID **Application** identity (certificate
+  name or SHA-1) passed to `codesign`
+- `SIGN_INSTALLER_IDENTITY` — the Developer ID **Installer** identity passed to
+  `pkgbuild` (optional; without it the `.pkg` itself is unsigned)
+- notarization credentials, either:
+  - `NOTARY_PROFILE` — a `notarytool store-credentials` keychain profile name, or
+  - `NOTARY_KEY_FILE` + `NOTARY_KEY_ID` + `NOTARY_ISSUER_ID` — an App Store
+    Connect API key (`.p8` file path, key id, issuer id)
+
+The app is signed inside-out under the hardened runtime with
+[`scripts/lattice.entitlements`](../scripts/lattice.entitlements) (JIT
+entitlements required by the embedded JavaScript runtime), then the `.app`,
+`.pkg`, and `.dmg` are each notarized and stapled, hard-failing unless Apple
+returns **Accepted**.
+
+The `Desktop Release` workflow does the same automatically when these repository
+secrets are configured (absent secrets — e.g. on forks — still produce working
+unsigned artifacts):
+
+| Secret                                                           | Contents                                                 |
+| ---------------------------------------------------------------- | -------------------------------------------------------- |
+| `MACOS_CERT_APP_P12` / `MACOS_CERT_APP_P12_PASSWORD`             | Developer ID Application cert (base64 `.p12`) + password |
+| `MACOS_CERT_INSTALLER_P12` / `MACOS_CERT_INSTALLER_P12_PASSWORD` | Developer ID Installer cert (base64 `.p12`) + password   |
+| `MACOS_KEYCHAIN_PASSWORD`                                        | password for the ephemeral build keychain                |
+| `MACOS_APP_IDENTITY` / `MACOS_INSTALLER_IDENTITY`                | the two signing identity names                           |
+| `NOTARY_KEY_P8`                                                  | App Store Connect API key (base64 `.p8`)                 |
+| `NOTARY_KEY_ID` / `NOTARY_ISSUER_ID`                             | the API key's id + issuer id                             |
+
 ## Limitations
 
 - **Image processing (`sharp`) and `sqlite-vec` acceleration are unavailable** in

package/docs/retrieval.md

@@ -92,6 +92,47 @@ Opt-in per-table approximate-nearest-neighbor index built from the stored vector
 `doctor` reports). Requires the extension server-side (pgvector) or loaded
 (sqlite-vec).
 
+Once built, the index is **kept in sync with writes** — you don't rebuild it by
+hand after every change. On Postgres each insert/update/delete mirrors the row
+into the index incrementally (on the same fire-and-forget path as the embedding
+write), and `refreshEmbeddings` reconciles the index after a bulk backfill. As a
+universal safety net across backends, `search()` **verifies the index is in sync
+with the stored vectors before trusting it** (a cheap count-parity check); if the
+index has drifted it transparently falls back to the exact in-process scan over
+the source-of-truth store, so a stale index is never silently served — it only
+ever costs a slower query, never a wrong result. (Incremental per-row maintenance
+is currently Postgres-only; a `sqlite-vec` index falls back to the scan after a
+write until it is rebuilt — correct either way.)
+
+**Cloud members.** In a multi-member cloud, a scoped member has no grant on the
+internal embeddings store or the native index, so its `search()` / `hybridSearch()`
+reach the vectors only through a `SECURITY DEFINER` function that returns just the
+chunks for rows the member may see (filtered by the same row-visibility rule that
+governs every other read, keyed on the member's role) and scores them in-process.
+The member scan is exact and has no over-fetch by which a member could infer hidden
+rows; result rows are re-checked by row-level security on the base relation. Owners
+and local (non-cloud) callers are unaffected — the routing is automatic.
+
+**Tuning & operations.** The HNSW index can be tuned at build time via
+`embeddings.index = { m, efConstruction }` and per query via `search(..., { efSearch })`
+(`hybridSearch` too); all default to pgvector's own values, so omitting them builds
+and queries exactly as before. `lattice index status` shows per-table index health
+(dimension, params, build time, staleness) from an internal `__lattice_vector_index`
+registry; `lattice reindex <table>` rebuilds one table's index, and `lattice doctor
+--fix` rebuilds any index it reports stale. An auto-rebuild after a bulk
+`refreshEmbeddings` reuses the recorded build params.
+
+**Scale.** For larger corpora, `embeddings.index.quantization = 'halfvec'` (pgvector
+≥ 0.7) stores the index at 16-bit half precision — roughly halving its memory at a
+small recall cost — while the embeddings store stays full precision, so the exact
+scan fallback and any later full-precision rebuild are unaffected. Lattice runs
+against a **single** Postgres or SQLite database: there is no sharding, no
+replication, and no distributed index, and `search()` / `hybridSearch()` keep the
+`topK ≤ 1000` bound (`SEARCH_TOPK_MAX`) so one query can't fan out into a
+whole-table read. These are deliberate boundaries — when you need billions of
+vectors, horizontal scale-out, or a managed vector service, run a dedicated vector
+database alongside Lattice and keep Lattice as the system of record.
+
 > **v4.2 — bounded retrieval reads.** `search()` / `hybridSearch()` clamp the
 > caller's `topK` (`clampTopK`, `SEARCH_TOPK_MAX = 1000`) **before** the indexed
 > arm over-fetches `topK * N` candidates, so a single large `topK` can't fan out