@happyvertical/smrt-core 0.40.70 → 0.41.0

package/AGENTS.md

@@ -213,8 +213,8 @@ Production DDL comes from the **manifest** paths
 **registry** paths feed `getTestDatabase()` and emit foreign-key indexes
 production never gets: the suite runs on a richer schema than it ships.
 
-- Change column/index emission on every shipping path, proven by a path-parity
-  test (#2359 adds one). A "same as migrations" comment is a claim to check.
+- Change column/index emission on every shipping path, proven by the path-parity
+  test `src/schema/schema-path-parity.test.ts` (#2359; index rules in the module doc). A "same as migrations" comment is a claim to check.
 - Every new query predicate ships with its index, or a reason it doesn't.
 - Numeric types, uuid casts, conflict targets, timestamps, migrations: run the
   `test:postgres` lane — SQLite affinity accepts what PostgreSQL rejects.

package/agents/change-feed.md

@@ -13,4 +13,4 @@ Adapter-agnostic change-observation spine (`src/change-feed.ts`) — the server
 - `getChangesSince(db, { since, tables?, tenantId?, limit? }) → { changes, cursor, resyncRequired?, resyncCursor? }`: strictly monotonic cursor; polling with returned cursors misses no committed change and never repeats one. A cursor that cannot be served incrementally — pruned below the retained `[floor..horizon]` run, or foreign/ahead of the horizon — gets `resyncRequired: true` with empty `changes`, an unadvanced `cursor`, and `resyncCursor` set to the current horizon so clients can full-refetch then resume incrementally; detection runs on the UNFILTERED log so `tables`/`tenantId` filters never trigger or mask it. `getTenantScopedChangesSince()` resolves tenant via the DispatchBus resolver hook (fail-closed: tenancy on + no context → global rows only; tenant `T` sees `T` + global rows, never another tenant).
 - `getTableVersion(db, table) → number`: the per-table change version (`MAX(seq)` for the table, replica-stable — no per-process divergence), the ETag source for zero-query conditional GETs (#1765). Advances on any framework write to the table (CRUD and sync-apply, which all `save()`/`delete()`). A table with no retained entry of its own falls back to the global horizon (never a resettable low value) so an all-pruned table cannot false-304 a stale client; only 0 when the feed is empty.
 - Generated `_changes` routes: REST (`GET {basePath}/_changes`, requires `authMiddleware`, otherwise 401 — per-model `api.public` does NOT apply) and SvelteKit (`{routesDir}/_changes/+server.ts`, requires an authenticated principal on `locals`; opt out via `sveltekit.changesRoute.enabled: false`). Query params: `since`, `tables` (comma-separated), `limit`. Responses stay HTTP 200 in the resync state — `resyncRequired` is protocol state, not an error, and `resyncCursor` is the resume cursor after the client completes a full refetch.
-- Retention: `pruneChangeFeed(db, { maxAgeMs?, maxRows? })` — schedule it. Pruning deletes oldest-first and always retains the newest entry (a non-empty feed is never emptied), which is what makes pruned-cursor detection provable and keeps caught-up consumers polling normally. Raw-SQL writes are invisible to the feed (same documented gap as the #1499 cache); `bumpChangeFeed(db, { table, rowId? })` is the manual escape hatch.
+- Retention: `pruneChangeFeed(db, { maxAgeMs?, maxRows?, dryRun? })` — scheduled since #2375 by `runRetentionSweep()` (30-day default), so nothing needs to call it directly; `dryRun` counts the same predicate instead of deleting. Pruning deletes oldest-first and always retains the newest entry (a non-empty feed is never emptied), which is what makes pruned-cursor detection provable and keeps caught-up consumers polling normally. Raw-SQL writes are invisible to the feed (same documented gap as the #1499 cache); `bumpChangeFeed(db, { table, rowId? })` is the manual escape hatch.

package/agents/schema-paths.md

@@ -35,18 +35,124 @@ Production DDL takes the manifest route:
                         use; no in-repo caller outside its own tests)
 ```
 
-The suite takes the registry route, and the registry route emits indexes the
-manifest route does not — per-column foreign-key indexes, and STI partial FK
-indexes filtered by `_meta_type`. Tests therefore run against a richer schema
-than any deployment receives. `src/testing/database.ts`'s "Generate schema using
-SchemaGenerator (same as migrations)" comment describes an intent, not the code.
-
-The manifest STI path even populates a `fkColumnsByClass` map and never reads it
-— only its registry counterpart iterates one — and the manifest CTI path has no
-FK loop at all. Both manifest paths then skip the explicit
-`@foreignKey(X, { indexed: true })` opt-in — the CTI one under a comment claiming
-"FK columns and unique columns get their own indexes", which holds on the
-registry paths and not on this one.
+The suite takes the registry route. Before #2359 the registry route emitted
+indexes the manifest route did not — per-column foreign-key indexes, and STI
+partial FK indexes filtered by `_meta_type` — so tests ran against a richer
+schema than any deployment received, the manifest STI path populated a
+`fkColumnsByClass` map it never read, and the manifest CTI path had no FK loop
+at all. `src/testing/database.ts`'s "same as migrations" comment described an
+intent, not the code.
+
+Since #2359 the two families share one set of index helpers and
+`src/schema/schema-path-parity.test.ts` runs the same fixture manifest through
+the manifest paths, through `ObjectRegistry.registerFromManifest()` + the
+registry paths, and through `getAllSchemasAsDefinitions()`, asserting identical
+column and index sets. Extend that fixture with every generator change; a
+divergence is a bug in the generator, not an exception to add to the test.
+
+### Index rules (#2359)
+
+- **Reference columns are always indexed.** `ensureReferenceColumnIndexes()`
+  runs last on every path and gives each `@foreignKey`, `@crossPackageRef` and
+  tenant column `<table>_<column>_idx` unless an UNQUALIFIED index (no `WHERE`,
+  no JSON path) already leads with it — the `conflictColumns` unique index or an
+  `indexed: true` opt-in, or the column's own inline UNIQUE. A partial
+  `WHERE _meta_type = …` index does not count: base-class polymorphic queries
+  carry no discriminator predicate. `indexed: true` on a reference column is
+  redundant. Roll the index wave out to production with
+  `smrt db:migrate --postgres-safe` (concurrent-index mode, #2362): a plain
+  atomic batch takes SHARE/ACCESS EXCLUSIVE locks for ~230 index builds. STI FK indexes are plain, one per
+  column, not per-class partial.
+- **No index on the primary key.** `<table>_id_idx` is gone from every path,
+  and `conflictColumns` equal to the PK column set emit no conflict index
+  (`ON CONFLICT (id)` binds to the PK constraint). `SchemaComparer` drops the
+  legacy non-unique single-column PK index from existing databases without
+  `--drop-indexes` when the live table reports that column as its sole primary
+  key (never a UNIQUE one — on PostgreSQL that may back a custom-named PRIMARY
+  KEY constraint, and `DROP INDEX` on it would fail the atomic batch).
+- **Slug loading keeps its index.** Custom `conflictColumns` replace the
+  `(slug, context)` unique index; `loadFromSlug()`/`getId()`/`getSavedId()`
+  still filter on slug/context, so a plain `<table>_slug_context_idx` is kept
+  (additive; routing those lookups through the conflict key would change which
+  row a slug resolves to). The tenant-led default key below counts as serving
+  it (`servesSlugLookup()`): a tenant-scoped slug lookup carries the tenant
+  predicate (#2365) and is served by the prefix, so no second index.
+- **Tenant-scoped tables key per tenant (#2360).** A tenant-scoped class with
+  no explicit `conflictColumns` upserts on, and indexes,
+  `(tenant_id, slug, context)` — `(tenant_id, slug, context, _meta_type)` for
+  an STI hierarchy — resolved by one rule on both paths:
+  `ManifestGenerator.normalizeConflictColumns()` materializes it into
+  `decoratorConfig.conflictColumns` for the manifest paths (so the manifest,
+  the schema, `smrt-knowledge.json` and the runtime read one value), and
+  `ObjectRegistry.getConflictColumns()` derives the same value at runtime from
+  the schema owner's `tenantScoped` config (`ObjectRegistry.getTenantColumn()`;
+  an STI child resolves through its root; a `@report` class through its
+  group/bucket columns; a custom primary key through that key). Explicit
+  `conflictColumns` are never rewritten. `src/schema/conflict-target.ts` holds
+  the shared helpers. Consequences: the index NAME stays
+  `<table>_slug_context_idx` / `_slug_context_meta_type_idx`, so the differ
+  swaps the columns of an existing global unique in place by name (a superset
+  key — creating it cannot fail on existing rows); the tenant-led key also
+  serves the tenant column, so `<table>_tenant_id_idx` is no longer emitted
+  for those tables (an existing one is an orphan the differ drops only with
+  `--drop-indexes`); NULL-tenant rows (`mode: 'optional'` outside a tenant
+  context) dedup among themselves through the SDK's null-aware upsert
+  (`IS NOT DISTINCT FROM` under a PostgreSQL advisory lock / an in-process
+  lock on SQLite) — application-enforced now, where the old global index was
+  database-enforced: the tenant-led index treats NULLs as distinct, so raw SQL
+  can insert two global rows with one slug, and a raw
+  `ON CONFLICT (slug, context…)` against such a table no longer binds (use
+  `WHERE NOT EXISTS`, plus an advisory lock on PostgreSQL). Emitting
+  `NULLS NOT DISTINCT` on PostgreSQL ≥ 15 (the SDK already detects it) would
+  restore the database arbiter — a follow-up. The `save()` path serializes an
+  unset tenant field as an explicit `NULL` whatever its registered type,
+  because the SDK rejects an upsert whose conflict column is missing from the
+  row.
+- **Rolling the tenant-led key out (#2360).** There is no mixed-version state:
+  new code against the old index fails every NEW-object create on a
+  tenant-scoped default-key table (PostgreSQL 42P10, SQLite "ON CONFLICT
+  clause does not match…"), and old code against the new index fails the same
+  way, because the conflict target must match the unique index's column set
+  exactly; only persisted objects (upsert on `id`) keep saving. Deploy the code
+  and run `smrt db:migrate` in the same maintenance step. The plan is one
+  `DROP INDEX` + `CREATE UNIQUE INDEX` per table under the SAME name (a
+  superset key, so the build cannot fail when the old same-name index was a
+  valid UNIQUE over the subset key; a #1165-class table whose old index was
+  non-unique or missing may hold duplicates that a superset UNIQUE rejects —
+  `db:diff` shows which tables' old index is non-unique or missing; dedupe
+  those rows before migrating). Atomic mode swaps every table in one
+  transaction: `DROP INDEX` takes ACCESS EXCLUSIVE and holds it until commit,
+  which blocks ALL access to those tables — reads included — for the batch;
+  size `statementTimeout` for the largest tenant-scoped table. That is the
+  maintenance window this rollout requires anyway (no mixed-version state), so
+  run this wave — the #2359 index wave included — in atomic mode inside it;
+  the "roll out with `--postgres-safe`" advice above applies to a #2359-only
+  wave, because `--postgres-safe` runs the two statements sequentially per
+  table, so each table has NO conflict index between them and a failed rebuild
+  leaves it without one until the re-run. The recreate has no automatic
+  DOWN: reverting the code means re-creating the old index by hand. And
+  legacy NULL-tenant rows fork rather than get adopted — a tenant-context save
+  whose slug matches a `(NULL, slug, ctx)` row now inserts `(tenant, slug,
+  ctx)` beside it, and that tenant no longer sees the legacy row — so backfill
+  `tenant_id` (anytown: `SET tenant_id = context::uuid`) BEFORE this release.
+  Ingestion that relied on natural-key dedup across tenants now inserts one
+  row per tenant (release note).
+- **STI `@field({ unique: true })` is enforced through indexes** (the differ can
+  add an index to an existing table, never a column constraint): a full
+  `<table>_<col>_unique_idx` when the STI base declares it, one
+  `<table>_<col>_<class>_unique_idx WHERE _meta_type = '<qualified>'` per class
+  when only descendants do — uniqueness per concrete class, not across the
+  subtree. DuckDB/JSON have no partial indexes, so the descendant-scoped shape
+  (`isStiSubtypeUniqueIndex`) is not emitted there — degrading it to a full
+  UNIQUE would constrain every subtype; the DDL strategy and the differ both
+  skip it, while other partial indexes keep degrading to full ones as before. Remember the
+  framework serializes an unset text field as `''`, so a unique optional text
+  field must be `nullable: true` with a `null` initializer or every unset row
+  collides.
+- **Every class in an STI hierarchy carries the schema of the one shared
+  table**, generated from the root base (`ManifestGenerator.generateSchemas()`
+  resolves the root through `findSTIBaseInfo`), so a child never treats its own
+  descendant-only unique field as base-declared.
 
 `src/schema/utils.ts` sits in between, and the two exports differ:
 
@@ -94,10 +200,10 @@ epic's fixes land.
 ### 1. Verify against the production path, not the test path
 
 Any change to column or index emission goes on **all** paths that ship and is
-proven by a path-parity test. #2359 adds that test under `src/schema/`; until it
-lands, assert the parity yourself in the nearest generator test — a green suite
-otherwise proves the registry paths only. Read the call graph before believing a
-comment: "same as migrations" was wrong for years.
+proven by the path-parity test (`src/schema/schema-path-parity.test.ts`, #2359)
+— extend its fixture; a green suite otherwise proves the registry paths only.
+Read the call graph before believing a comment: "same as migrations" was wrong
+for years.
 
 ### 2. Every new query predicate ships with its index
 
@@ -129,17 +235,21 @@ every package; do not sample a few and extrapolate.
 ### 5. Index intent belongs on both the constraint and the read path
 
 A conflict target is not automatically a unique index, and a unique index is not
-automatically the index a read path uses. Custom `conflictColumns` replace the
-`(slug, context)` index while `loadFromSlug`/`getId` still query slug+context;
-STI drops `@field({ unique: true })`. Check the pair, not the declaration.
+automatically the index a read path uses. Custom `conflictColumns` used to
+replace the `(slug, context)` index while `loadFromSlug`/`getId` still queried
+slug+context, and STI dropped `@field({ unique: true })` — both fixed in #2359,
+see "Index rules" above. Check the pair, not the declaration.
 
 ### 6. Multi-tenancy is a whole-path property
 
 Every unique constraint and every conflict target on a tenant-scoped table
 includes the tenant column — otherwise a second tenant's `save()` of the same
-natural key updates the first tenant's row through `DO UPDATE SET` (#2360). And
-every read path is interceptor-aware: hydration (`loadFromId`/`loadFromSlug`),
-get-by-slug, vector search, and collection memory, not only `list()` (#2365).
+natural key updates the first tenant's row through `DO UPDATE SET` (#2360; the
+default key now does, see "Index rules" — an explicit `conflictColumns` that
+omits the tenant column is the class author's own key and is not rewritten).
+And every read path is interceptor-aware: hydration
+(`loadFromId`/`loadFromSlug`), get-by-slug, vector search, and collection
+memory, not only `list()` (#2365).
 
 ### 7. Retry only transient errors
 
@@ -183,10 +293,10 @@ production down on rollout (#2362).
 
 ### 13. Composite indexes are declared, not inferred (#2357)
 
-The generated set only covers foreign keys, unique/conflict columns,
-`updated_at`, the STI discriminator, `tenant_id`, and single columns opted in
-with `@field({ indexed: true })`. A list workload's access path is composite,
-so declare it:
+The generated set only covers foreign keys, unique/conflict columns, the STI
+discriminator, reference columns (#2359), the default list ordering (rule 18
+below), and single columns opted in with `@field({ indexed: true })`. A list
+workload's access path is composite, so declare it:
 
 ```ts
 @smrt({
@@ -203,9 +313,10 @@ scans a btree either way, so an ascending index also serves the matching
 `ORDER BY ... DESC` as an ordered scan with no Sort node. `unique` and `where`
 (partial index) are honoured.
 
-`appendDeclaredIndexes()` runs on all five entry points, before
-`ensureTenantIdIndex()`, so a declared composite leading with the tenant column
-replaces the automatic standalone `tenant_id` index rather than duplicating it.
+`appendDeclaredIndexes()` runs first on all five entry points, ahead of
+`ensureDefaultListOrderingIndex()` (rule 18) and `ensureReferenceColumnIndexes()`,
+so a declared composite leading with the tenant column (or any reference column)
+replaces the automatic standalone index rather than duplicating it.
 Unknown columns, malformed entries, and a name collision with a different index
 all fail generation — a silently dropped index only surfaces later as a
 production slowdown. Rule 8 above is why this works at runtime at all.
@@ -326,3 +437,350 @@ warning, as `db:migrate` drops them. Do not add a new consumer of the
 string, and do not write a private CREATE INDEX renderer — the retired ones
 dropped `where` and `jsonPath` (#2358). Every DDL strategy also spells out
 `PRIMARY KEY NOT NULL`: SQLite lets a bare non-INTEGER PRIMARY KEY hold NULL.
+
+### 17. The merged table shape is registration-order independent (#2372)
+
+`getAllSchemas()` and `getAllSchemasAsDefinitions()` fold every class that
+shares a physical table — the whole STI hierarchy — into one shape. Both route
+through `buildMergedTableSchemas()`, which groups contributors by table and
+then merges them in a **deterministic** order: the STI base first, then
+ancestors before descendants, then by qualified name.
+
+That order matters because the first contributor seeds the table: it supplies
+the fallback base columns, the `idType`, the conflict columns and the cached
+DDL, and its columns win every merge conflict. When registration order decided
+it, an STI child that carries no manifest `schema` — the external- and
+consumer-manifest case — seeded the table from bare fallback columns and the
+base class's richer ones were skipped when it registered later, yielding
+`context TEXT` instead of `context TEXT NOT NULL DEFAULT ''` and timestamps
+with no NOT NULL/DEFAULT. The shipped content manifest lists `Article` before
+`Content`, so the losing order was the one that shipped, and the differ
+compares types only, so the weak fresh-create was never repaired.
+
+Two invariants keep the two assembly paths agreeing:
+
+- `createBaseColumns()` mirrors what `generateSchemaFromManifest` /
+  `generateSTISchemaFromManifest` emit for the same table, so a table built
+  from runtime field metadata alone has the same NOT NULL/DEFAULT shape as one
+  built from a manifest. Note `_meta_type` is `TEXT NOT NULL` with **no**
+  default, matching the generator.
+- `fieldsToColumns()` reads `required`, `default`, and `description` from the
+  top level *or* `_meta`. Registry fields normalize them into `_meta`
+  (`manifest-field-merge.ts`), so reading only the top level silently dropped
+  NOT NULL and DEFAULT for every registry-sourced field.
+
+STI columns stay nullable regardless of the field's `required` flag
+(`fieldsToColumns(fields, { stiUnionColumns: true })`): the table holds the
+union of all subtypes' fields, so a column only one subtype declares is never
+populated on a sibling's row. Declared defaults are still emitted. This matches
+`generateSTISchemaFromManifest`, which sets `notNull: false` on every non-system
+STI column.
+
+When adding a class-level input to the merged shape, take it from the seeding
+contributor rather than "whichever class arrives first", and cover it with a
+child-first/base-first equality test.
+
+### 18. The generator owns the index for its own default ordering (#2363)
+
+Every generated list surface — REST, MCP, the SvelteKit list route — pages with
+`ORDER BY created_at DESC, <pk> ASC` (`DEFAULT_LIST_ORDER_BY`, #2367), and
+until #2363 no schema path indexed `created_at` (the dead AST path indexed
+`updated_at`), so the framework's own default page was a sequential scan plus a
+top-N sort. `ensureDefaultListOrderingIndex()` now runs on all five paths and
+emits:
+
+- `(<tenant column>, created_at)` on a tenant-scoped table — the tenancy
+  interceptor puts `tenant_id = ?` in front of every list, so the tenant column
+  leads and `created_at` orders within it. This composite **replaces** the
+  standalone tenant index from #2359: a B-tree serves every prefix of its
+  column list, so `ensureDefaultListOrderingIndex()` is called first and
+  `ensureReferenceColumnIndexes()` then sees the column as already served. The
+  tenant column is found by `referenceKind === 'tenantId'`, never by the
+  `tenant_id` spelling — `@smrt({ tenantScoped: { field } })` renames it.
+- `(created_at)` otherwise.
+
+Three deliberate omissions, so nobody "fixes" them later:
+
+- **No `DESC`.** `IndexDefinition` carries no per-column direction and
+  PostgreSQL scans a B-tree backwards just as cheaply.
+- **No primary-key tiebreak column.** The default order mixes directions
+  (`created_at DESC, id ASC`), so no single-direction index satisfies the whole
+  key; the leading columns already turn a full sort into an index scan plus an
+  incremental sort over rows sharing a timestamp.
+- **Not scoped per STI subtype.** `(_meta_type, created_at)` would serve a
+  child collection's list but not the base class's polymorphic one, which
+  carries no discriminator predicate — the same reasoning that keeps STI
+  reference indexes plain (#2359). One unqualified index per shared table.
+
+An existing UNQUALIFIED index that already leads with the same columns
+suppresses it — a partial or JSON-path index never counts. That is how a
+declared `@smrt({ indexes: [...] })` composite (#2357) takes over: declaring
+`(tenant_id, created_at, status)` replaces the generated pair, while declaring
+a different sort column such as `(tenant_id, publish_date)` sits **beside** it,
+because that index cannot order the default page. Declared indexes are appended
+before this helper for exactly that reason; anything that appends an index in
+future goes in the same slot, ahead of `ensureDefaultListOrderingIndex()` and
+`ensureReferenceColumnIndexes()`.
+
+### 19. One conflict-target rule, applied on every producer
+
+`save()` upserts on `ObjectRegistry.getConflictColumns()`; the schema must
+carry exactly one unique index over those columns (or they must be the
+primary key). Keep the derivation in `src/schema/conflict-target.ts` and let
+every producer call it — the three manifest pipelines share
+`ManifestGenerator.applyGenerationPasses()` since #2360 because
+`ManifestBuilder` had silently skipped the report passes for months. When you
+add a way for the key to vary (a new decorator option, a new class kind),
+thread it through `getConflictColumns()`, `normalizeConflictColumns()` and the
+generator's `resolveConflictTarget()` together, and extend the parity test's
+"unique index == conflict target" assertion; a key the runtime uses and the
+schema does not index is a hard PostgreSQL error (42P10) on the first save,
+and a key the schema indexes without the tenant column is the silent
+cross-tenant overwrite this rule exists for.
+
+### 20. Every generated index name is length-guarded before it leaves a path (#2374)
+
+PostgreSQL truncates any identifier past 63 **bytes** and reports nothing;
+SQLite and DuckDB do not, so the entire test suite was blind to it. The 66-byte
+`content_contribution_revisions_contribution_id_revision_number_idx` shipped
+that way — only the differ's signature-equivalence check kept it from emitting
+`add_index` on every run. Two names agreeing for 63 bytes is the real hazard:
+`CREATE INDEX IF NOT EXISTS` no-ops against the wrong index, and the second
+index is never created.
+
+`schema/index-utils.ts` owns the guard, and it splits by who owns the name:
+
+- **Generated index, trigger and PL/pgSQL function names** →
+  `shortenIdentifier()`. Deterministic `<head>_<digest><suffix>`, digest taken
+  over the **full** original so a shared prefix still yields distinct names, and
+  a recognised suffix (`_idx`, `_unique_idx`, `_key`, `_pkey`) preserved.
+- **Hand-declared `@smrt({ indexes: [{ name }] })`** → `assertIdentifierFits()`,
+  a hard error in `validateDeclaredIndex()`. Renaming what a developer wrote is
+  worse than refusing it, and `SchemaComparer` matches indexes **by name**
+  first, so a 70-byte declaration could never match the 63-byte index
+  PostgreSQL stored and `db:migrate` would emit `add_index` forever.
+- **Table and column names** → deliberately **not** guarded. PostgreSQL
+  truncates identifiers *consistently on every reference*: `CREATE TABLE
+  "<80 bytes>"` and a later `SELECT ... FROM "<the same 80 bytes>"` both resolve
+  to the same stored 63-byte name, so one long name round-trips fine end to end.
+  `smrt-users` depends on this — it ships an intentional 80-byte
+  `@smrt({ tableName })` (`permission_policy_table_name_that_is_far_too_long…`)
+  and derives unique Postgres RLS policy names from it. An earlier revision of
+  this rule hard-errored here on the theory that the runtime resolves tables by
+  name and would break; that theory is wrong for the reason above, and the error
+  broke `packages/users`. The residual collision risk is over a name the
+  developer chose, not one the generator manufactured.
+
+`enforceIdentifierLimits()` is the single call site per path, placed **after**
+`ensureReferenceColumnIndexes()` — nothing may lengthen a name after it. Doing
+the shortening at the end rather than at each `indexes.push()` is safe because
+the digest covers the whole original name, so entries distinct before shortening
+stay distinct after; the helper still throws if two ever collide. The migrate
+leg's `withConflictIndex()` (`registry/schema-builder.ts`) and the PostgreSQL
+trigger-function name call `shortenIdentifier()` directly, because they compose
+a name outside the generator's index list. Note that an over-long *table* name
+still yields in-limit, distinct *index* names, because the shortening runs over
+the whole composed name.
+
+The digest is FNV-1a, not `node:crypto`: `index-utils.ts` is re-exported from
+`schema/utils.ts`, which exists to keep Node built-ins out of browser bundles.
+It only has to be *stable* — a shortened name that changed between releases
+would make every deployment drop and recreate the index — so the parity and
+unit tests pin the literal output rather than recomputing it. Unpaired
+surrogates are folded to U+FFFD before both counting and hashing, so the digest
+is taken over exactly the bytes the driver transmits.
+
+Existing databases migrate **by name swap, without a rebuild**: the live index
+still carries the name PostgreSQL truncated it to, the manifest now carries the
+shortened one, and the differ claims it by signature (columns + uniqueness +
+predicate), emitting nothing — including under `includeDroppedIndexes`. See
+`migrations/__tests__/index-drift.test.ts` and the PostgreSQL lane test
+`schema/issue-2374-identifier-length-postgres.optional.test.ts`.
+
+Out of scope, deliberately: constraint names PostgreSQL invents for itself. A
+CTI table's inline `UNIQUE` produces an implicit `<table>_<column>_key`, which
+can exceed 63 bytes even when the table and column each fit. SMRT never names
+it, and PostgreSQL disambiguates its own truncations by appending a counter
+rather than collapsing them, so there is no silent-collision hazard there.
+
+### 21. The `_smrt_` prefix does not mean "system table" (#2376)
+
+`bootstrapSystemTables()` owns nine hand-written tables; ~25 more `_smrt_*`
+tables belong to `@smrt()` models and are created by `db:migrate` (feature
+flags, prompt overrides, subscription plans, report schedules, field policies,
+jobs). Never classify by prefix — use `SYSTEM_TABLE_NAMES`
+(`schema/system-table-shapes.ts`, derived from the DDL parse) plus
+`FRAMEWORK_OPERATIONAL_TABLES` / `RETIRED_SYSTEM_TABLES` in `system/schema.ts`.
+The change-feed writer skipped by prefix, so clients syncing those domain
+tables through `_changes` never saw an update.
+
+Editing `ALL_SYSTEM_TABLES` requires bumping `SMRT_SCHEMA_VERSION` *and*
+appending to `SMRT_SCHEMA_DDL_CHECKSUMS` — the version gates the DDL replay, so
+without a bump no existing database ever applies the change. A new **column**
+additionally needs an `addColumnIfMissing()` entry in `system/compatibility.ts`
+(`CREATE TABLE IF NOT EXISTS` is a no-op on an existing table).
+`system-schema-evolution.test.ts` enforces both, and asserts a legacy database
+upgrades to exactly the shape a fresh install gets.
+
+`_smrt_jobs` / `_smrt_job_events` are dual-owned: `db:migrate` creates them,
+the compatibility pass reshapes them. On a fresh install bootstrap runs first,
+so their pass is deferred — `ensureDeferredSystemTableCompatibility()` re-runs
+until the tables exist, then stamps a `<version>+deferred-compat` marker. It
+runs OUTSIDE the bootstrap lock and swallows its own failures: those statements
+target tables the framework does not own, and inside the PostgreSQL transaction
+one failure would roll back system-table creation with it. Only
+`ensureBootstrapSystemTableCompatibility()` (the tables the DDL itself creates)
+belongs inside the lock.
+
+Reconciling `_smrt_jobs.task_id` uniqueness reads the live index catalog, which
+is implemented for PostgreSQL and SQLite only; DuckDB and the JSON adapter keep
+the redundant compat index rather than risk dropping the one that enforces the
+upsert conflict target. When reading a PostgreSQL catalog array, cast it
+(`attname::text`) and parse both shapes — a driver with no parser registered for
+the array OID returns the raw `{a,b}` literal, and reading that as "no columns"
+silently inverts an index-existence decision.
+
+## Referential integrity lives in `delete()`, not in the DDL
+
+No schema path emits a `FOREIGN KEY` clause on any engine — grep `schema/ddl/*`,
+`schema-manager.ts` and `differ.ts` for `REFERENCES` and you get nothing. That is
+a deliberate position, not an oversight: emitting constraints changes delete
+semantics for every consumer, requires topological table ordering in both
+migrate paths (neither orders today), and needs a plan for the orphans already in
+production databases. Emitting them is tracked separately.
+
+What `@foreignKey(..., { onDelete })` therefore means is *application* behaviour,
+applied by `SmrtObject.delete()` through `src/cascade.ts` (#2371):
+
+| Reference | Default when `onDelete` is absent |
+|---|---|
+| Column is part of the referencing class's `conflictColumns`, and is not a `@tenantId()` field | `CASCADE` |
+| Polymorphic `(metaType, metaId)` association row | `CASCADE` |
+| Anything else, including every `@tenantId()` field | `NO ACTION` — the row is left alone |
+
+The natural-key rule is what cleans junction rows up without any per-package
+annotation: a junction declares
+`@smrt({ conflictColumns: ['content_id', 'asset_id', 'relationship'] })`, so the
+row is *identified* by the content and cannot outlive it. An ordinary child
+(`Order.customerId`) is keyed by `(slug, context)` and keeps its pre-#2371
+behaviour unless it opts in explicitly.
+
+**`@tenantId()` is excluded even though it lands in `conflictColumns`.**
+#2360 leads every tenant-scoped class's *default* natural key with the
+tenant column, so without this exclusion, deleting one `Tenant` row would
+recursively CASCADE through every tenant-scoped table in the schema that has
+not declared its own `conflictColumns` — the overwhelming majority. The
+tenant column scopes ownership; it does not identify the row the way a
+junction's foreign key does. Detected via the `__tenancy.isTenantIdField`
+marker on `FieldMeta` (`smrt-core` reads it structurally so it never depends
+on `smrt-tenancy`). `@tenantId()` exposes no `onDelete` option today, so
+this cannot currently be overridden per field — found in review before this
+landed (originally reachable, untested, and undocumented).
+
+Properties to keep if you touch that module:
+
+- **The plan is registry-derived and rebuilt per delete.** Registration is
+  incremental — manifests load lazily and tests register classes between cases —
+  so a cached plan would silently skip a table that registered later. Cache it
+  only behind an invalidation hook that every registration path calls.
+- **A class with nothing pointing at it skips the transaction entirely — but
+  `CascadePlan.isEmpty` requires no polymorphic association class anywhere in
+  the process, not just no typed references.** `buildCascadePlan()` pushes
+  *every* registered `SmrtPolymorphicAssociation` subclass into
+  `plan.polymorphic` unconditionally (`cascade.ts` around
+  `isPolymorphicAssociationClass`): a `metaType` column can point at any class
+  at runtime, so there is no static metadata to scope it by the target being
+  deleted. One registered polymorphic class anywhere makes `isEmpty` false for
+  every delete in that process — do not read "the common case skips the
+  transaction" as "most deletes in a real app skip it"; in a multi-package app
+  that registers even one polymorphic association, almost none do.
+  `runCascadeDelete()` builds the plan for `getResolvedQualifiedName()` (not the
+  bare constructor name — two packages can register the same simple name).
+- **Cascaded rows are removed set-based.** Their `beforeDelete`/`afterDelete`
+  hooks and interceptors do not run and no change-feed tombstone is written for
+  them, which is exactly what a DB-level `ON DELETE CASCADE` does. Only the
+  object `delete()` was called on runs the lifecycle. Do not "improve" this into
+  a per-row model delete without deciding what that means for sync consumers.
+- **Everything is one transaction where the adapter has one**, including the
+  object's own `DELETE`, whenever there is anything to cascade. The `RESTRICT`
+  checks run first, before any mutation, so a refusal costs nothing; the
+  transaction is what makes a refusal *deeper* in the graph safe.
+- **`_smrt_embeddings` and `_smrt_contexts` are matched by id *and* a
+  class-name candidate set, not id alone.** Their class columns store the
+  *runtime* constructor name, which for an STI hierarchy is a concrete
+  subclass rather than the class the cascade planned from — id-alone matching
+  looked STI-safe, but let two unrelated classes using `idType: 'text'`
+  (non-UUID, not guaranteed globally unique) collide on a shared id value and
+  delete each other's rows (review fix). `ownerClassCandidates()` expands to
+  every STI hierarchy member of the class the ids actually belong to, in both
+  qualified and simple form. A failure to clean them is logged, never raised —
+  an application database may predate the table, and losing derived rows must
+  not fail a valid delete.
+
+`_smrt_changes`, `_smrt_ai_usage`, `_smrt_signals` and the dispatch tables are
+deliberately **not** cascaded. They are append-only logs; the change feed in
+particular receives the delete's own tombstone, so cascading it would erase the
+record that tells sync clients the row is gone.
+
+### 22. System tables get a retention policy, not just a prune function (#2375)
+
+Four framework-owned tables grow with traffic and nothing used to remove a row:
+`_smrt_changes` (one per save/delete), `_smrt_ai_usage` (one per AI call, and
+persistence is on by default), `_smrt_contexts` (whose `expires_at` nothing
+enforced) and `_smrt_dispatch` (an operator-only `dispatch:cleanup`).
+`src/system/retention.ts` is now the single place that bounds them.
+
+- **`runRetentionSweep(db, policy)` is the entry point.** It runs the four
+  built-in tasks in a fixed order, then every task other packages contributed
+  via `registerRetentionTask()` — `@happyvertical/smrt-jobs` registers
+  `_smrt_jobs`/`_smrt_job_events`, `@happyvertical/smrt-users` registers
+  session/magic-link/CLI-auth expiry. A task that throws is recorded on its own
+  result and the sweep continues; a missing table reports `unavailable`, so a
+  sweep is safe against a partially bootstrapped database.
+- **A contributed task only exists in a process that loaded its package.** Both
+  packages register on import from their entry point, and the registry lives on
+  `globalThis` (like `ObjectRegistry`) so a duplicated `smrt-core` resolution
+  cannot split it. `smrt db:prune` optionally imports both packages for exactly
+  this reason — a project that installs neither correctly gets neither task.
+- **Defaults are opt-out, not opt-in.** `DEFAULT_RETENTION_POLICY` covers the
+  four built-in tables (changes 30 days, AI usage 90 days, dispatch 30 days
+  completed / 90 days failed, contexts strictly by their own `expires_at`), and
+  those are the ones `smrt.configure({ retention })` tunes. Contributed tasks
+  carry their own defaults and their own window options —
+  `DEFAULT_JOB_RETENTION` (7 days terminal / 30 days failed / 30 days events,
+  set through `registerJobRetentionTasks()` or the runner's `retention.jobs`),
+  and expired credentials, which have no window because an expired credential
+  has nothing worth retaining. Every task, built-in or contributed, can be
+  turned off: a table set to `false`, a task set to `false` under `tasks`, or
+  `enabled: false` for the whole sweep — through `smrt.configure`,
+  `smrt db:prune --skip`, or the runner's `retention` config.
+- **Contributed task names are prefixed with the owning package's short name**
+  (`jobs-records`, `jobs-events`, `users-sessions`, …) because the registry is
+  one process-global namespace.
+- **Scheduling lives outside core.** A running `TaskRunner` sweeps every six
+  hours (`retention: false` opts out) and `smrt db:prune` is the cron entry
+  point. The first runner sweep is one interval after `start()`, never at
+  start: a crash-looping worker must not become a delete loop.
+- **Every prune counts before it deletes.** `rowCount` is not reliably
+  populated across the engines SMRT supports, so counting is both what gives a
+  usable figure and what lets `dryRun` preview the *same* predicate rather than
+  an approximation of it. Count and delete are two statements and deliberately
+  not one transaction — a maintenance pass must not hold a write lock over a
+  large delete — so the figure is approximate under concurrent writers. Where
+  two bounds can select the same row (`pruneChangeFeed`, `pruneAiUsage`), the
+  second bound excludes what the first already accounted for, so a dry run does
+  not count an entry twice.
+- **Every retention predicate ships with its index** (rule 2 applies to
+  maintenance SQL too): `_smrt_contexts(expires_at)`,
+  `_smrt_ai_usage(tenant_id, created_at)` — which is also the subscriptions
+  billing meter's range scan — `_smrt_dispatch(status, processed_at)` and
+  `(status, updated_at)` come from the system DDL, so they reach existing
+  databases through the `SMRT_SCHEMA_VERSION` bump that replays it.
+  `_smrt_jobs(status, completed_at)` comes from
+  `ensureJobsSystemTableCompatibility()` instead, because `_smrt_jobs` is
+  generated from a decorated class and does not exist yet when bootstrap runs;
+  the jobs collection calls that path on every `initialize()`.
+- **Expiry enforcement is prune-side only.** `recall()`/`recallAll()` keep
+  their documented "expiry is not applied at read time" contract — changing it
+  would change read semantics for existing callers, which is a different issue
+  from bounding storage. `LearningMemory` filters expired rows itself.

package/dist/browser.js

@@ -3,13 +3,13 @@ import { DEFAULT_AI_COST_RATES, estimateAiUsageCost } from "./adapters/cost-rate
 import { MetricsAdapter } from "./adapters/metrics.js";
 import { PubSubAdapter } from "./adapters/pubsub.js";
 import "./adapters/index.js";
+import { classifyDatabaseError, classifyDialectMessage, isAbortedTransactionError, isDeterministicDatabaseError, isNotNullViolationError, isTransientDatabaseError, isUniqueViolationError } from "./db-errors.js";
+import { AIError, ConfigurationError, DatabaseError, ErrorUtils, FilesystemError, NetworkError, RuntimeError, SmrtError, TenantIsolationError, ValidationError, ValidationReport, ValidationUtils } from "./errors.js";
 import { config } from "./config.js";
 import { SignalSanitizer } from "./signals/sanitizer.js";
 import { SignalBus } from "./signals/bus.js";
-import { assertPostgresSystemTimestampsCurrent, ensureDispatchSubscriptionsSystemTableCompatibility, ensureDispatchSystemTableCompatibility, ensureJobEventsSystemTableCompatibility, ensureJobsSystemTableCompatibility, ensureLegacySystemTableCompatibility, getDatabaseEngine, migratePostgresSystemTimestamps, planPostgresSystemTimestampMigrations, tableExists } from "./system/compatibility.js";
+import { assertPostgresSystemTimestampsCurrent, ensureBootstrapSystemTableCompatibility, ensureDeferredSystemTableCompatibility, ensureDispatchSubscriptionsSystemTableCompatibility, ensureDispatchSystemTableCompatibility, ensureJobEventsSystemTableCompatibility, ensureJobsSystemTableCompatibility, ensureLegacySystemTableCompatibility, getDatabaseEngine, migratePostgresSystemTimestamps, planPostgresSystemTimestampMigrations, tableExists } from "./system/compatibility.js";
 import { SmrtClass } from "./class.js";
-import { classifyDatabaseError, classifyDialectMessage, isAbortedTransactionError, isDeterministicDatabaseError, isNotNullViolationError, isTransientDatabaseError, isUniqueViolationError } from "./db-errors.js";
-import { AIError, ConfigurationError, DatabaseError, ErrorUtils, FilesystemError, NetworkError, RuntimeError, SmrtError, TenantIsolationError, ValidationError, ValidationReport, ValidationUtils } from "./errors.js";
 import { convertTypeToJsonSchema, generateToolFromMethod, generateToolManifest, shouldIncludeMethod } from "./tools/tool-generator.js";
 import { executeToolCall, executeToolCalls, formatToolResults, validateToolCall } from "./tools/tool-executor.js";
 import { SmrtObject } from "./object.js";
@@ -20,6 +20,7 @@ import { SmrtHierarchical } from "./hierarchical.js";
 import { SmrtJunction } from "./junction.js";
 import { SmrtPolymorphicAssociation } from "./polymorphic-association.js";
 import "./signals/index.js";
+import { DEFAULT_RETENTION_POLICY, clearRetentionTasks, getRetentionTasks, pruneAiUsage, pruneExpiredContexts, registerRetentionTask, runRetentionSweep, unregisterRetentionTask } from "./system/retention.js";
 import "./system/index.js";
 import "./tools/index.js";
-export { AIError, AiUsageCollector, AiUsagePersistenceHandler, ConfigurationError, DEFAULT_AI_COST_RATES, DatabaseError, ErrorUtils, FilesystemError, MetricsAdapter, NetworkError, ObjectRegistry, PubSubAdapter, RuntimeError, SMRT_COLLECTION_BASE_NAMES, SignalBus, SignalSanitizer, SmrtClass, SmrtCollection, SmrtError, SmrtHierarchical, SmrtJunction, SmrtObject, SmrtPolymorphicAssociation, TenantIsolationError, ValidationError, ValidationReport, ValidationUtils, assertPostgresSystemTimestampsCurrent, classifyDatabaseError, classifyDialectMessage, config, convertTypeToJsonSchema, ensureDispatchSubscriptionsSystemTableCompatibility, ensureDispatchSystemTableCompatibility, ensureJobEventsSystemTableCompatibility, ensureJobsSystemTableCompatibility, ensureLegacySystemTableCompatibility, estimateAiUsageCost, executeToolCall, executeToolCalls, formatToolResults, generateToolFromMethod, generateToolManifest, getDatabaseEngine, isAbortedTransactionError, isDeterministicDatabaseError, isNotNullViolationError, isSmrtCollectionExtendsName, isTransientDatabaseError, isUniqueViolationError, migratePostgresSystemTimestamps, planPostgresSystemTimestampMigrations, shouldIncludeMethod, smrt, smrt as smrtRegistry, tableExists, validateToolCall };
+export { AIError, AiUsageCollector, AiUsagePersistenceHandler, ConfigurationError, DEFAULT_AI_COST_RATES, DEFAULT_RETENTION_POLICY, DatabaseError, ErrorUtils, FilesystemError, MetricsAdapter, NetworkError, ObjectRegistry, PubSubAdapter, RuntimeError, SMRT_COLLECTION_BASE_NAMES, SignalBus, SignalSanitizer, SmrtClass, SmrtCollection, SmrtError, SmrtHierarchical, SmrtJunction, SmrtObject, SmrtPolymorphicAssociation, TenantIsolationError, ValidationError, ValidationReport, ValidationUtils, assertPostgresSystemTimestampsCurrent, classifyDatabaseError, classifyDialectMessage, clearRetentionTasks, config, convertTypeToJsonSchema, ensureBootstrapSystemTableCompatibility, ensureDeferredSystemTableCompatibility, ensureDispatchSubscriptionsSystemTableCompatibility, ensureDispatchSystemTableCompatibility, ensureJobEventsSystemTableCompatibility, ensureJobsSystemTableCompatibility, ensureLegacySystemTableCompatibility, estimateAiUsageCost, executeToolCall, executeToolCalls, formatToolResults, generateToolFromMethod, generateToolManifest, getDatabaseEngine, getRetentionTasks, isAbortedTransactionError, isDeterministicDatabaseError, isNotNullViolationError, isSmrtCollectionExtendsName, isTransientDatabaseError, isUniqueViolationError, migratePostgresSystemTimestamps, planPostgresSystemTimestampMigrations, pruneAiUsage, pruneExpiredContexts, registerRetentionTask, runRetentionSweep, shouldIncludeMethod, smrt, smrt as smrtRegistry, tableExists, unregisterRetentionTask, validateToolCall };