@ai-matrx/data 0.23.0 → 0.24.0

package/CHANGELOG.md

@@ -1,5 +1,25 @@
 # Changelog
 
+## 0.24.0 — doors hardened (independent review)
+
+- **Every write on a versioned table is guarded.** `remove` (and `archive` / `restore`) take the same
+  `{ expectedVersion }` / `{ overwrite: true }` guard as `update`; `expectedVersion` on an UNversioned
+  table is refused (it used to be ignored silently). `remove` returns `{ status }` (or a conflict).
+- **Reads exclude soft-deleted rows** by default (`read`, `listAll`, `page`, `count`); `includeDeleted: true` opts in.
+- **`listAll` / `page` order by the full primary key** (stable paging on composite keys).
+- **Types:** ops infer `TableDescriptor<Row, Insert, Update>` directly, so an insert error names the
+  table's Insert type; `read`/`listAll`/`page` take `columns: [...]` and type the result to them; the
+  `Where` callback receives a typed `Filter<Row>` (column names and value types checked).
+- **Pay-only runtime tables:** `--flags-index` writes `<schema>.flags.ts` per schema plus
+  `flags-index.ts` loaders; `loadTableByName(flagLoaders, "schema.table")` loads only that schema.
+- **The description ships in the package** (`description/*.json.gz`, 0.5 MB): `matrx-data generate`
+  needs no repository or database access (`--from` still takes a fresher one). README documents it.
+- `matrx-data emit` accounts for every callable-door register row: `index.json` →
+  `report.registeredDoorsOutsideDescription` by reason, and the emit fails if any row is unexplained.
+
+Consumer action: `remove(db, t, id)` on a versioned table now needs a guard and returns `{ status }`;
+`read(db, t, id, "cols")` becomes `read(db, t, id, { columns: [...] })`. No consumer used 0.23.0's doors yet.
+
 ## 0.23.0 — the doors: one way to talk to the database
 
 - **`matrx-data` CLI** (`bin`): `matrx-data emit [--out db-contract] [--check]` reads the LIVE database

package/README.md

@@ -32,6 +32,40 @@ npm install @ai-matrx/data
 | `createActiveOrgCookie` | ONE browser memory of the organization a person is acting in, shared by every Matrx surface on the apex (`matrx-active-org`, `Domain=.aimatrx.com`): identity-keyed (`<userId>:<orgId>`), domain-wide only on a listed apex, host-only elsewhere. Stores the stored-selection rung of the canonical order; never decides. |
 | `uniqueChannelTopic` | Realtime channel hygiene: unique per-mount topics so React 19 double-invoked effects and Fast Refresh can never collide with a still-joined channel. |
 
+## The doors — one way to talk to the database
+
+Every app talks to the database through doors over descriptors it **generates locally** from the
+ONE database description. The description ships inside this package (`description/`, gzipped, the one
+this version was built with), so no repository or database access is needed.
+
+```bash
+# in your app (package.json: "db:generate": "matrx-data generate --out src/db/generated --schemas workbench,chat")
+npx matrx-data generate --out src/db/generated --schemas workbench          # one file per schema
+npx matrx-data generate --out src/db/generated --schemas workbench --check  # CI: fail when stale
+npx matrx-data generate --out src/db/generated --flags-index                # + per-schema flags for tableByName
+# fresher than this package version? With the direct-DB variables (SUPABASE_MATRIX_*):
+npx matrx-data emit --out db-contract && npx matrx-data generate --from db-contract --out src/db/generated
+```
+
+```ts
+import { createDb, listAll, read, insert, update, remove, call } from "@ai-matrx/data/db";
+import { noteFolders, notes, noteFolderGetOrCreate } from "./db/generated/workbench";
+
+const db = createDb({ client: supabase, organization: () => activeOrgId }); // wraps YOUR client
+const pinned = await listAll(db, notes, (q) => q.eq("is_pinned", true));       // complete; typed filter; live rows
+const row = await read(db, notes, id, { columns: ["id", "label"] });           // typed to those columns
+await insert(db, notes, { folder_id, label });         // child: the folder's organization, never the active one
+await update(db, notes, id, { label }, { expectedVersion: row.version });      // or { overwrite: true }
+await remove(db, notes, id, { expectedVersion: v });   // soft delete; archive-only tables refuse
+await call(db, noteFolderGetOrCreate, { p_name: "Clients" });
+// @ai-matrx/api: pass db.credentials as its `credentials` option — one sign-in feeds both lanes
+```
+
+Rules come from the description (the live table registry), never the call site: organization by the
+access ladder; reads never filter by the active organization and exclude soft-deleted rows unless
+`includeDeleted`; `listAll` is complete and ordered by the full primary key; every write on a versioned
+table is guarded; the door set follows the registry type; `createServiceDb` is server-only.
+
 ## `@ai-matrx/data/net`
 
 | Export | What it does |

package/bin/emit.mjs

@@ -137,12 +137,32 @@ function collect(schemas) {
         and (${ROLES.map((r) => `has_function_privilege('${r}', p.oid, 'EXECUTE')`).join(" or ")})
     ) f`);
 
+  // Every register row, matched to its live function (schema-qualified type names normalized, so
+  // "p_row block_states" matches "p_row platform.block_states"), and when it is not in the
+  // description, WHY — so no register row is ever unaccounted for.
   const doors = readJson(`
     select coalesce(json_agg(json_build_object(
-      'schema', d.schema_name, 'name', d.function_name, 'identity', d.identity_args,
+      'schema', d.schema_name, 'name', d.function_name, 'identity', coalesce(p.ident, d.identity_args),
       'gatePredicate', d.gate_predicate, 'anonymousCallers', d.anonymous_callers, 'signedInCallers', d.signed_in_callers,
-      'nonClientLane', d.non_client_lane, 'argumentRules', d.argument_rules, 'refusalOnly', d.refusal_only
-    )), '[]') from platform.client_callable_door d`);
+      'nonClientLane', d.non_client_lane, 'argumentRules', d.argument_rules, 'refusalOnly', d.refusal_only,
+      'outside', case
+        when p.oid is null and not exists (select 1 from pg_proc p2 join pg_namespace n2 on n2.oid = p2.pronamespace
+                                            where n2.nspname = d.schema_name and p2.proname = d.function_name) then 'function no longer exists'
+        when p.oid is null then 'no live overload has the registered signature'
+        when not (d.schema_name = any(${S})) then 'schema not exposed to PostgREST (server-only)'
+        when not (${ROLES.map((r) => `has_function_privilege('${r}', p.oid, 'EXECUTE')`).join(" or ")}) then 'no client role may execute it'
+        when p.prorettype in ('trigger'::regtype, 'event_trigger'::regtype) then 'trigger function'
+        when exists (select 1 from pg_depend dd where dd.classid = 'pg_proc'::regclass and dd.objid = p.oid and dd.deptype = 'e') then 'extension-owned'
+      end
+    )), '[]')
+    from platform.client_callable_door d
+    left join lateral (
+      select p.oid, p.prorettype, pg_get_function_identity_arguments(p.oid) as ident
+      from pg_proc p join pg_namespace n on n.oid = p.pronamespace
+      where n.nspname = d.schema_name and p.proname = d.function_name
+        and regexp_replace(pg_get_function_identity_arguments(p.oid), '\\m[a-z_][a-z0-9_]*\\.', '', 'g')
+          = regexp_replace(d.identity_args, '\\m[a-z_][a-z0-9_]*\\.', '', 'g')
+      limit 1) p on true`);
 
   const enums = readJson(`
     select coalesce(json_object_agg(n.nspname || '.' || t.typname,
@@ -262,7 +282,7 @@ export function buildDescription() {
   const doorsByKey = new Map(doors.map((d) => [doorKey(d.schema, d.name, d.identity), d]));
 
   const files = {};
-  const index = { format: FORMAT, schemas: {}, publications: {}, report: { tablesWithoutRegistryRow: [], untypedRegistryRows: [], registeredDoorsOutsideDescription: 0 } };
+  const index = { format: FORMAT, schemas: {}, publications: {}, report: { tablesWithoutRegistryRow: [], untypedRegistryRows: [], registeredDoorsOutsideDescription: {} } };
   const publications = {};
   for (const { pub, table } of published) {
     const name = table.replace(/^realtime\.messages_\d{4}_\d{2}_\d{2}$/, "realtime.messages");
@@ -313,9 +333,14 @@ export function buildDescription() {
   }
   for (const k of Object.keys(publications)) publications[k].sort();
   index.publications = publications;
-  // Register rows whose function is outside the exposed schemas, not executable by any client role,
-  // or whose identity no longer matches a live signature.
-  index.report.registeredDoorsOutsideDescription = doors.length - matchedDoors;
+  // Every register row not in the description, by reason (each named; none unaccounted for).
+  const outside = {};
+  for (const d of doors) if (d.outside) (outside[d.outside] ??= []).push(`${d.schema}.${d.name}(${d.identity})`);
+  for (const k of Object.keys(outside)) outside[k].sort();
+  index.report.registeredDoorsOutsideDescription = outside;
+  index.report.registeredDoors = { total: doors.length, inDescription: matchedDoors, outside: doors.length - matchedDoors };
+  const unexplained = doors.length - matchedDoors - Object.values(outside).reduce((n, l) => n + l.length, 0);
+  if (unexplained !== 0) throw new Error(`${unexplained} register row(s) neither in the description nor explained — the emitter's matching is wrong`);
   index.report.tablesWithoutRegistryRow.sort();
   index.report.untypedRegistryRows.sort();
   index.contractSha256 = sha256(Object.keys(index.schemas).sort().map((s) => `${s}:${index.schemas[s].sha256}`).join("\n"));

package/bin/generate.mjs

@@ -17,6 +17,8 @@
  */
 import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
 import path from "node:path";
+import { fileURLToPath } from "node:url";
+import { gunzipSync } from "node:zlib";
 
 const RESERVED = new Set(
   "break case catch class const continue debugger default delete do else enum export extends false finally for function if import in instanceof new null return super switch this throw true try typeof var void while with yield let static implements interface package private protected public await".split(
@@ -157,29 +159,56 @@ export function renderSchema(desc, schemaSha, importFrom) {
   return out;
 }
 
-export function renderFlagsIndex(descs, importFrom) {
+/** One schema's flags (`<schema>.flags.ts`) — loaded only when a runtime name names that schema. */
+export function renderSchemaFlags(desc, importFrom) {
   const index = {};
-  for (const d of descs) for (const [name, t] of Object.entries({ ...d.tables, ...d.views })) index[`${d.schema}.${name}`] = tableDescriptor(d.schema, name, t);
-  return `/** GENERATED by \`matrx-data generate --flags-index\` — the flags of every table, for tableByName(). Never edit by hand. */
+  for (const [name, t] of Object.entries({ ...desc.tables, ...desc.views })) index[`${desc.schema}.${name}`] = tableDescriptor(desc.schema, name, t);
+  return `/** GENERATED by \`matrx-data generate --flags-index\` — ${desc.schema}'s table flags for tableByName(). Never edit by hand. */
 /* eslint-disable */
 import type { FlagsIndex } from "${importFrom}";
 
-export const flagsIndex: FlagsIndex = ${JSON.stringify(index)} as unknown as FlagsIndex;
+export const flags: FlagsIndex = ${JSON.stringify(index)} as unknown as FlagsIndex;
 `;
 }
 
-export function generate({ from, out, schemas, check, importFrom, flagsIndex = process.argv.includes("--flags-index") }) {
-  const index = JSON.parse(readFileSync(path.join(from, "index.json"), "utf8"));
+/** Per-schema loaders: `loadTableByName(flagLoaders, "schema.table")` imports only that schema's flags. */
+export function renderFlagLoaders(schemas, importFrom) {
+  return `/** GENERATED by \`matrx-data generate --flags-index\` — never edit by hand. */
+/* eslint-disable */
+import type { FlagLoaders } from "${importFrom}";
+
+export const flagLoaders: FlagLoaders = {
+${schemas.map((s) => `  ${key(s)}: () => import("./${s}.flags").then((m) => m.flags),`).join("\n")}
+};
+`;
+}
+
+/** The description ships inside @ai-matrx/data (gzipped); `--from` points at a fresher one. */
+export const BUNDLED_DESCRIPTION = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "description");
+
+function readDescriptionFile(dir, name) {
+  const plain = path.join(dir, name);
+  if (existsSync(plain)) return readFileSync(plain, "utf8");
+  const gz = `${plain}.gz`;
+  if (existsSync(gz)) return gunzipSync(readFileSync(gz)).toString("utf8");
+  throw new Error(`${name} not found in ${dir} (neither ${name} nor ${name}.gz)`);
+}
+
+export function generate({ from = BUNDLED_DESCRIPTION, out, schemas, check, importFrom, flagsIndex = process.argv.includes("--flags-index") }) {
+  const index = JSON.parse(readDescriptionFile(from, "index.json"));
   const names = schemas?.length ? schemas : Object.keys(index.schemas).sort();
   const files = {};
   const descs = [];
   for (const s of names) {
     if (!index.schemas[s]) throw new Error(`schema "${s}" is not in the description (${Object.keys(index.schemas).length} schemas)`);
-    const desc = JSON.parse(readFileSync(path.join(from, `${s}.json`), "utf8"));
+    const desc = JSON.parse(readDescriptionFile(from, `${s}.json`));
     descs.push(desc);
     files[`${s}.ts`] = renderSchema(desc, index.schemas[s].sha256, importFrom);
   }
-  if (flagsIndex) files["flags-index.ts"] = renderFlagsIndex(descs, importFrom);
+  if (flagsIndex) {
+    for (const d of descs) files[`${d.schema}.flags.ts`] = renderSchemaFlags(d, importFrom);
+    files["flags-index.ts"] = renderFlagLoaders(descs.map((d) => d.schema), importFrom);
+  }
   if (check) {
     const stale = Object.entries(files).filter(([n, t]) => !existsSync(path.join(out, n)) || readFileSync(path.join(out, n), "utf8") !== t).map(([n]) => n);
     if (stale.length) {

package/bin/matrx-data.mjs

@@ -3,7 +3,9 @@
  * matrx-data — @ai-matrx/data's build-time CLI.
  *
  *   matrx-data emit     [--out <dir>] [--check]      live database → the database description (read-only)
- *   matrx-data generate --from <dir> --out <dir> [--schemas a,b]   description → typed doors in THIS app
+ *   matrx-data generate --out <dir> [--from <dir>] [--schemas a,b] [--flags-index] [--check]
+ *                       description → typed doors in THIS app (--from defaults to the description
+ *                       shipped inside this package version)
  *
  * The package ships the runtime and this CLI; each app generates its own doors
  * locally (Prisma's model), so a new column never waits on a package release.
@@ -25,12 +27,12 @@ async function main() {
     const { generate } = await import("./generate.mjs");
     const from = flag("--from");
     const out = flag("--out");
-    if (!from || !out) {
-      console.error("matrx-data generate: --from <description dir> and --out <dir> are required");
+    if (!out) {
+      console.error("matrx-data generate: --out <dir> is required (--from defaults to the description shipped in this package)");
       return 2;
     }
     return generate({
-      from: path.resolve(from),
+      ...(from ? { from: path.resolve(from) } : {}),
       out: path.resolve(out),
       schemas: flag("--schemas")?.split(",").map((s) => s.trim()).filter(Boolean),
       check: rest.includes("--check"),

package/dist/db.cjs

@@ -53,6 +53,7 @@ __export(db_exports, {
   isMissingSessionError: () => isMissingSessionError,
   isRecordUnavailableError: () => isRecordUnavailableError,
   listAll: () => listAll,
+  loadTableByName: () => loadTableByName,
   mergeJsonColumn: () => mergeJsonColumn,
   page: () => page,
   parseCookieHeader: () => parseCookieHeader,
@@ -1054,18 +1055,26 @@ function byPk(q, t, id) {
   const key = id;
   return t.pk.reduce((acc, col) => acc.eq(col, key[col]), q);
 }
-async function read(db, table, id, columns = "*") {
+var selectOf = (columns) => columns && columns.length ? columns.join(",") : "*";
+function live(q, table, includeDeleted) {
+  return table.softDelete && !includeDeleted ? q.is("deleted_at", null) : q;
+}
+function ordered(q, table, orderBy, ascending = true) {
+  const cols = orderBy ? [orderBy, ...table.pk.filter((c) => c !== orderBy)] : table.pk.length ? table.pk : ["id"];
+  return cols.reduce((acc, c) => acc.order(c, { ascending }), q);
+}
+async function read(db, table, id, opts = {}) {
   assertDoor(db, table, "read");
-  return check(`read ${qualified(table)}`, await byPk(from(db, table).select(columns), table, id).maybeSingle());
+  const q = live(byPk(from(db, table).select(selectOf(opts.columns)), table, id), table, opts.includeDeleted);
+  return check(`read ${qualified(table)}`, await q.maybeSingle());
 }
 async function listAll(db, table, where, opts = {}) {
   assertDoor(db, table, "read");
-  const order = opts.orderBy ?? table.pk[0] ?? "id";
   return readAllRows(
     ({ from: lo, to: hi }) => {
-      let q = from(db, table).select(opts.columns ?? "*", { count: "exact" });
+      let q = live(from(db, table).select(selectOf(opts.columns), { count: "exact" }), table, opts.includeDeleted);
       if (where) q = where(q);
-      return q.order(order, { ascending: true }).range(lo, hi);
+      return ordered(q, table, opts.orderBy).range(lo, hi);
     },
     {
       label: `listAll ${qualified(table)}`,
@@ -1076,14 +1085,14 @@ async function listAll(db, table, where, opts = {}) {
 }
 async function page(db, table, window2) {
   assertDoor(db, table, "read");
-  let q = from(db, table).select(window2.columns ?? "*", { count: "exact" });
+  let q = live(from(db, table).select(selectOf(window2.columns), { count: "exact" }), table, window2.includeDeleted);
   if (window2.where) q = window2.where(q);
-  const res = await q.order(window2.orderBy ?? table.pk[0] ?? "id", { ascending: window2.ascending ?? true }).range(window2.from, window2.to);
+  const res = await ordered(q, table, window2.orderBy, window2.ascending ?? true).range(window2.from, window2.to);
   return { rows: check(`page ${qualified(table)}`, res) ?? [], total: res.count ?? null };
 }
-async function count(db, table, where) {
+async function count(db, table, where, opts = {}) {
   assertDoor(db, table, "read");
-  let q = from(db, table).select("*", { count: "exact", head: true });
+  let q = live(from(db, table).select("*", { count: "exact", head: true }), table, opts.includeDeleted);
   if (where) q = where(q);
   const res = await q;
   check(`count ${qualified(table)}`, res);
@@ -1116,6 +1125,24 @@ async function insert(db, table, values) {
   const stored = check(`insert ${qualified(table)}`, res);
   return many ? stored : stored[0];
 }
+async function writeRow(db, table, id, values, door, guard) {
+  const expected = guard && "expectedVersion" in guard ? guard.expectedVersion : void 0;
+  if (expected !== void 0 && !table.versioned) {
+    throw new DoorRefusedError(door, "expectedVersion on a table that is not versioned \u2014 nothing would guard it; pass no guard");
+  }
+  if (table.versioned && expected === void 0 && !(guard && "overwrite" in guard && guard.overwrite)) {
+    throw new DoorRefusedError(door, "a versioned table takes { expectedVersion } or { overwrite: true }");
+  }
+  if (expected !== void 0) {
+    return await guardedUpdate({
+      expectedVersion: expected,
+      applyUpdate: ({ expectedVersion, nextVersion }) => byPk(from(db, table).update({ ...values, version: nextVersion }), table, id).eq("version", expectedVersion).select("*").maybeSingle(),
+      fetchCurrent: () => byPk(from(db, table).select("*"), table, id).maybeSingle()
+    });
+  }
+  const row = check(door, await byPk(from(db, table).update(values), table, id).select("*").maybeSingle());
+  return row ? { status: "saved", row } : { status: "not_found" };
+}
 async function update(db, table, id, patch, guard) {
   assertDoor(db, table, "update");
   const door = `update ${qualified(table)}`;
@@ -1124,29 +1151,30 @@ async function update(db, table, id, patch, guard) {
     const blocked = table.readOnlyColumns.filter((c) => c in values);
     if (blocked.length) throw new DoorRefusedError(door, `read-only to clients: ${blocked.join(", ")}`);
   }
-  if (table.versioned && guard && "expectedVersion" in guard) {
-    return guardedUpdate({
-      expectedVersion: guard.expectedVersion,
-      applyUpdate: ({ expectedVersion, nextVersion }) => byPk(from(db, table).update({ ...values, version: nextVersion }), table, id).eq("version", expectedVersion).select("*").maybeSingle(),
-      fetchCurrent: () => byPk(from(db, table).select("*"), table, id).maybeSingle()
-    });
-  }
-  if (table.versioned && !(guard && "overwrite" in guard && guard.overwrite)) {
-    throw new DoorRefusedError(door, "a versioned table takes { expectedVersion } or { overwrite: true }");
-  }
-  const row = check(door, await byPk(from(db, table).update(values), table, id).select("*").maybeSingle());
-  return row ? { status: "saved", row } : { status: "not_found" };
+  return writeRow(db, table, id, values, door, guard);
 }
-async function remove(db, table, id) {
+async function remove(db, table, id, guard) {
   assertDoor(db, table, "remove");
   const door = `remove ${qualified(table)}`;
   if (table.softDelete) {
-    const row = check(door, await byPk(from(db, table).update({ deleted_at: (/* @__PURE__ */ new Date()).toISOString() }), table, id).select("*").maybeSingle());
-    return row ? "soft_deleted" : "not_found";
+    const res = await writeRow(db, table, id, { deleted_at: (/* @__PURE__ */ new Date()).toISOString() }, door, guard);
+    return res.status === "saved" ? { status: "soft_deleted" } : res.status === "conflict" ? { status: "conflict", currentVersion: res.currentVersion } : res;
   }
   if (table.archive) throw new DoorRefusedError(door, "this table archives instead of deleting \u2014 use archive()");
-  const rows = check(door, await byPk(from(db, table).delete(), table, id).select("*"));
-  return rows && rows.length ? "deleted" : "not_found";
+  const expected = guard && "expectedVersion" in guard ? guard.expectedVersion : void 0;
+  if (expected !== void 0 && !table.versioned) throw new DoorRefusedError(door, "expectedVersion on a table that is not versioned");
+  if (table.versioned && expected === void 0 && !(guard && "overwrite" in guard)) {
+    throw new DoorRefusedError(door, "a versioned table takes { expectedVersion } or { overwrite: true }");
+  }
+  let q = byPk(from(db, table).delete(), table, id);
+  if (expected !== void 0) q = q.eq("version", expected);
+  const rows = check(door, await q.select("*"));
+  if (rows && rows.length) return { status: "deleted" };
+  if (expected !== void 0) {
+    const current = check(door, await byPk(from(db, table).select("version"), table, id).maybeSingle());
+    if (current) return { status: "conflict", currentVersion: current.version };
+  }
+  return { status: "not_found" };
 }
 async function archive(db, table, id, opts = {}) {
   assertDoor(db, table, "archive");
@@ -1156,9 +1184,9 @@ async function archive(db, table, id, opts = {}) {
   else if (table.archive === "is_archived") values = { is_archived: !opts.restore };
   else if (table.softDelete && opts.restore) values = { deleted_at: null };
   else throw new DoorRefusedError(door, "the table has no archive column");
-  return check(door, await byPk(from(db, table).update(values), table, id).select("*").maybeSingle());
+  return writeRow(db, table, id, values, door, opts.guard);
 }
-var restore = (db, table, id) => archive(db, table, id, { restore: true });
+var restore = (db, table, id, guard) => archive(db, table, id, guard ? { restore: true, guard } : { restore: true });
 async function call(db, fn, args) {
   const door = `call ${fn.schema}.${fn.name}`;
   const role = db.role === "service" ? "service_role" : "authenticated";
@@ -1172,6 +1200,12 @@ function tableByName(index, name) {
   if (!t) throw new DoorRefusedError(`tableByName ${name}`, "no such table in the database description (schema-qualify it: schema.table)");
   return t;
 }
+async function loadTableByName(loaders, name) {
+  const schema = name.slice(0, name.indexOf("."));
+  const load = schema ? loaders[schema] : void 0;
+  if (!load) throw new DoorRefusedError(`tableByName ${name}`, "no such schema in the database description (schema-qualify it: schema.table)");
+  return tableByName(await load(), name);
+}
 function changesOf(table) {
   if (!table.realtime) throw new DoorRefusedError(`changesOf ${qualified(table)}`, "the table is not in the supabase_realtime publication");
   return { schema: table.schema, table: table.name };