@happyvertical/smrt-core 0.40.69 → 0.41.0

package/AGENTS.md

@@ -17,6 +17,7 @@ subsystem you are editing. This file keeps what holds across all of them.
 | `src/change-feed.ts` | the adapter-agnostic change-observation spine — `_smrt_changes`, cursors, table versions, generated `_changes` routes, retention | [agents/change-feed.md](agents/change-feed.md) |
 | `src/change-signals.ts` + the generated `_events` SSE route | the push companion to the change feed — the signal bus, cross-replica fan-out, the SSE route, and its documented gaps | [agents/change-signals.md](agents/change-signals.md) |
 | `src/generators/` + `src/vite-plugin/web-collections.ts` | REST/CLI/MCP/web-collection generation, the `manifestHash` emission sites, and generated conditional-GET / ETag v2 semantics | [agents/generators.md](agents/generators.md) |
+| `src/schema/` | the five `SchemaGenerator` entry points, which two reach production, why schema drift stayed invisible, and the #2382 index/tenancy rules | [agents/schema-paths.md](agents/schema-paths.md) |
 
 ## SmrtObject Lifecycle
 
@@ -25,6 +26,7 @@ subsystem you are editing. This file keeps what holds across all of them.
 - `initialize()`: loads field initializers, applies option values (options override initializers), loads from DB if id/slug provided
 - `save()`: upsert with STI validation, interceptor execution, auto-embeddings. Persisted objects (`isPersisted` — set by DB hydration and successful saves) upsert on `['id']` so natural-key edits (e.g. slug renames) update in place; new objects upsert on the natural-key conflict columns for ingestion-style dedup (#1472)
 - `is(criteria)` / `do(instructions)` / `describe()`: AI operations via function calling. They inject the object's own `toPublicJSON()` (sensitive fields stripped) as a "content body" so the model reasons over the instance. Options: `includeData: false` skips injection (for callers that already curate the relevant fields into the instruction); `maxDataLength` overrides the truncation budget. Neither key is forwarded to `ai.message()`. (#1567)
+- `save()` error contract (#2366): unique/PK violation → `ValidationError` `VALIDATION_UNIQUE_CONSTRAINT`, NOT NULL → `VALIDATION_REQUIRED_FIELD`, both on the first attempt on every adapter; any other database failure → `DatabaseError` with the driver error on `cause`
 - `getSlug()`: auto-generates from name → title → label → id
 - `loadRelated(fieldName)`: lazy-loads relationships (cached in `_loadedRelationships` Map)
 
@@ -74,7 +76,7 @@ references). Re-adding either requires the query builder to support it first;
 `src/__tests__/issue-2276-where-contract.test.ts` executes every accepted
 operator against a database to keep the two in step.
 
-STI child collections auto-filter by `_meta_type`.
+STI child collections auto-filter by `_meta_type`. Query bounds — `LIMIT 1` on `get()`, the `limit`/`offset` parser, the `orderBy` whitelist and sensitive/permission refusals, and the deterministic generated-list ordering (#2367) — are in [agents/query-bounds.md](agents/query-bounds.md).
 
 ## Bounded Collection Read Plans
 
@@ -97,7 +99,7 @@ Two persistence primitives every `SmrtObject`/`SmrtCollection` inherits — load
 
 ## @smrt() Decorator Options
 
-Key options: `tableName`, `tableStrategy` ('cti'|'sti'), `conflictColumns`, `api`/`mcp`/`cli` (generation config), `ai` (callable methods), `hooks` (beforeSave/afterSave/beforeDelete/afterDelete), `embeddings` (auto-generate), `tenantScoped`, `agent`, `ui` (`{ icon, label, description }` — nav/help hints round-tripped through the manifest as plain data; `description` is the object-level seed for form-level help, #2046).
+Key options: `tableName`, `tableStrategy` ('cti'|'sti'), `conflictColumns`, `indexes` (declared multi-column indexes, #2357 — see "Schema paths"), `api`/`mcp`/`cli` (generation config), `ai` (callable methods), `hooks` (beforeSave/afterSave/beforeDelete/afterDelete), `embeddings` (auto-generate), `tenantScoped`, `agent`, `ui` (`{ icon, label, description }` — nav/help hints round-tripped through the manifest as plain data; `description` is the object-level seed for form-level help, #2046).
 
 Registration sets `SMRT_TABLE_NAME` static property (survives minification).
 
@@ -203,13 +205,36 @@ byte/provenance/path/content drift, skips scans and manifest writes, and still
 generates routes, types, registration, and virtual modules. Omit it for normal
 local development and watch mode.
 
+## Schema paths (#2382)
+
+Production DDL comes from the **manifest** paths
+(`generateSTISchemaFromManifest`/`generateCTISchemaFromManifest`, selected in
+`src/scanner/manifest-generator.ts` → registered `schema` → `db:migrate`). The
+**registry** paths feed `getTestDatabase()` and emit foreign-key indexes
+production never gets: the suite runs on a richer schema than it ships.
+
+- Change column/index emission on every shipping path, proven by the path-parity
+  test `src/schema/schema-path-parity.test.ts` (#2359; index rules in the module doc). A "same as migrations" comment is a claim to check.
+- Every new query predicate ships with its index, or a reason it doesn't.
+- Numeric types, uuid casts, conflict targets, timestamps, migrations: run the
+  `test:postgres` lane — SQLite affinity accepts what PostgreSQL rejects.
+- Read `dist/manifest.json`/regenerated schemas for what a decorator produced;
+  count across all packages instead of sampling.
+- Tenant scoping is whole-path: every unique constraint and conflict target on a
+  tenant-scoped table carries the tenant column, and every read path — not only
+  `list()` — is interceptor-aware.
+- Rolling indexes out is part of the change: a bulk `CREATE INDEX` batch needs
+  the bounded, concurrent migrate path (#2362, Gotchas), or it takes production
+  down on deploy.
+
 ## Gotchas
 
 - **Filesystem support is a lazy boundary (#1979)**: `SmrtClass` acquires `options.fs` adapters via `createFilesystemAdapter()` (`src/filesystem-loader.ts`), never a static `@happyvertical/files` import — the files SDK statically pulls @aws-sdk/client-s3 and reaches googleapis, and a static edge here would land it in every downstream SSR bundle. Node/tsx/vite-dev runtimes resolve it on first use; fully-bundled deployments import `@happyvertical/smrt-core/filesystem` at startup. Use `importOptionalDependency()` (`src/lazy-external.ts`) for any similar optional heavyweight dependency.
 - **Never override toJSON()** — handles STI discriminator + meta field extraction. Use `transformJSON()`
 - **Property init order**: TypeScript initializers run first, then `initialize()` applies option values (options win)
-- **No runtime schema creation**: application tables must be prepared explicitly via migrations/tooling; runtime only verifies and fails clearly
-- **Retry logic**: `db.get()` (3 retries, 250ms) and `db.upsert()` (3 retries, 500ms) have built-in retry
+- **No runtime schema creation**: application tables must be prepared explicitly via migrations/tooling; runtime verification is `tableExists()` only (`src/schema/table-verifier.ts`) — no column, type, or index check
+- **PostgreSQL migrate batches are always time-bounded (#2362)**: `MigrationTracker.applyAll({ atomic: true })` emits `SET LOCAL lock_timeout`/`statement_timeout` before any DDL, so a batch blocked on one table cannot hold its earlier locks indefinitely. `postgresSafe: true` adds concurrent-index mode — non-index DDL commits atomically, then index DDL runs `CONCURRENTLY` on a session pinned via `db.acquireSession()` (a pooled `db.query` would not keep the `SET` and the DDL on one connection). That mode is deliberately **not atomic**: unfinished index migrations are recorded `failed`, not `running`, and their `error_message` carries a `[smrt: concurrent-index phase 1 committed]` marker so a reconciling re-run resumes at the index build instead of replaying committed DDL. INVALID indexes are found via `pg_index.indisvalid` (`pg_indexes` reports them as present) and dropped before rebuild. Operational detail: `packages/cli/AGENTS.md`.
+- **Retry logic is transient-only (#2366)**: `db.get()`/`db.upsert()` retry 4× total (initial + 3), but `ErrorUtils.withRetry` classifies via the cause chain (`src/db-errors.ts`) and rethrows deterministic failures immediately — constraint violations, bad input syntax, missing tables, aborted PG tx (`25P02`). `@happyvertical/sql` stringifies the driver text into `context.originalError`, so **never match `error.message`**; use `classifyDatabaseError()` / `isUniqueViolationError()` / `isAbortedTransactionError()`.
 - **Field caching**: `_cachedFields` populated during `Collection.create()` — eliminates async `getFields()` per query
 - **Smart cloning**: arrays/objects shallow-cloned in property init to prevent aliasing (Issue #22)
 - **Table verification cache**: `isTableVerified(dbUrl, tableName)` avoids redundant `tableExists()` calls

package/README.md

@@ -425,7 +425,26 @@ Every `SmrtObject`/`SmrtCollection` can persist learned knowledge via `remember(
 | `AIError` | AI provider failures |
 | `ValidationError` | Field/object validation failures |
 | `RuntimeError` | General runtime failures |
-| `ErrorUtils` | Sanitization and formatting helpers |
+| `ErrorUtils` | Retry policy (`withRetry`, `isRetryable`) plus sanitization helpers |
+
+### Database error classification
+
+Driver errors reach the model layer wrapped by `@happyvertical/sql`, which
+stringifies the driver text into `context.originalError` — so the constraint
+wording never appears on `error.message`. Classify through the cause chain
+instead of matching messages.
+
+| Export | Description |
+|--------|------------|
+| `classifyDatabaseError` | Walks the cause chain and returns the kind plus `deterministic` / `retryable` |
+| `classifyDialectMessage` | Matches a single raw dialect message (the DuckDB fallback) |
+| `isUniqueViolationError` | Unique or primary-key violation anywhere in the chain |
+| `isNotNullViolationError` | NOT NULL violation anywhere in the chain |
+| `isAbortedTransactionError` | Statement issued inside an aborted PostgreSQL transaction (`25P02`) |
+| `isDeterministicDatabaseError` | A retry cannot change the outcome |
+| `isTransientDatabaseError` | Contention or availability; a retry may succeed |
+| `DatabaseErrorKind` | Union of classification kinds |
+| `DatabaseErrorClassification` | Structured result of `classifyDatabaseError` |
 
 ### Tools (AI Function Calling)
 

package/agents/change-feed.md

@@ -13,4 +13,4 @@ Adapter-agnostic change-observation spine (`src/change-feed.ts`) — the server
 - `getChangesSince(db, { since, tables?, tenantId?, limit? }) → { changes, cursor, resyncRequired?, resyncCursor? }`: strictly monotonic cursor; polling with returned cursors misses no committed change and never repeats one. A cursor that cannot be served incrementally — pruned below the retained `[floor..horizon]` run, or foreign/ahead of the horizon — gets `resyncRequired: true` with empty `changes`, an unadvanced `cursor`, and `resyncCursor` set to the current horizon so clients can full-refetch then resume incrementally; detection runs on the UNFILTERED log so `tables`/`tenantId` filters never trigger or mask it. `getTenantScopedChangesSince()` resolves tenant via the DispatchBus resolver hook (fail-closed: tenancy on + no context → global rows only; tenant `T` sees `T` + global rows, never another tenant).
 - `getTableVersion(db, table) → number`: the per-table change version (`MAX(seq)` for the table, replica-stable — no per-process divergence), the ETag source for zero-query conditional GETs (#1765). Advances on any framework write to the table (CRUD and sync-apply, which all `save()`/`delete()`). A table with no retained entry of its own falls back to the global horizon (never a resettable low value) so an all-pruned table cannot false-304 a stale client; only 0 when the feed is empty.
 - Generated `_changes` routes: REST (`GET {basePath}/_changes`, requires `authMiddleware`, otherwise 401 — per-model `api.public` does NOT apply) and SvelteKit (`{routesDir}/_changes/+server.ts`, requires an authenticated principal on `locals`; opt out via `sveltekit.changesRoute.enabled: false`). Query params: `since`, `tables` (comma-separated), `limit`. Responses stay HTTP 200 in the resync state — `resyncRequired` is protocol state, not an error, and `resyncCursor` is the resume cursor after the client completes a full refetch.
-- Retention: `pruneChangeFeed(db, { maxAgeMs?, maxRows? })` — schedule it. Pruning deletes oldest-first and always retains the newest entry (a non-empty feed is never emptied), which is what makes pruned-cursor detection provable and keeps caught-up consumers polling normally. Raw-SQL writes are invisible to the feed (same documented gap as the #1499 cache); `bumpChangeFeed(db, { table, rowId? })` is the manual escape hatch.
+- Retention: `pruneChangeFeed(db, { maxAgeMs?, maxRows?, dryRun? })` — scheduled since #2375 by `runRetentionSweep()` (30-day default), so nothing needs to call it directly; `dryRun` counts the same predicate instead of deleting. Pruning deletes oldest-first and always retains the newest entry (a non-empty feed is never emptied), which is what makes pruned-cursor detection provable and keeps caught-up consumers polling normally. Raw-SQL writes are invisible to the feed (same documented gap as the #1499 cache); `bumpChangeFeed(db, { table, rowId? })` is the manual escape hatch.

package/agents/query-bounds.md

@@ -0,0 +1,45 @@
+<!-- Module doc for packages/core/AGENTS.md. Linked from the Modules table there. -->
+
+# Query bounds (#2367)
+
+`src/query-bounds.ts` is the single parser every generated read surface uses for
+`limit`/`offset`. A bound is a non-negative integer or it is a client error:
+malformed input raises a 400-typed `QueryBoundsError` (a `ValidationError`, so
+`withRetry` never retries it and `normalizeTypedHttpError` renders it as a
+structured 400) instead of reaching the driver as `LIMIT NaN`. Oversized pages
+are **clamped** to `MAX_LIST_LIMIT` (1000) rather than rejected, and an explicit
+`0` means zero rows — it is not folded into `DEFAULT_LIST_LIMIT` (50).
+
+- `collection.get()` / `findOne()` / `findById()` emit `LIMIT 1`.
+- `collection.list()` validates `limit`/`offset` but applies **no implicit
+  default**: it is the framework's bulk-read primitive and relationship,
+  junction, hierarchy and `listByIds()` callers all expect every matching row, so
+  a framework-wide default would truncate correct queries. Applications opt in
+  per collection with `defaultListLimit` / `maxListLimit` (validated at
+  construction). The generated surfaces enforce the ceiling on their own
+  untrusted input regardless.
+- `orderBy` runs the same rail as `where` (#1540) and `select` (#1902): terms are
+  checked against the field whitelist and refused for `@field({ sensitive: true })`
+  and `@field({ readPermission })` columns — ordering is a comparison, and
+  `?orderBy=api_secret&limit=1` is an oracle over a column the request may neither
+  filter on nor project — and for fields that are registered but **not
+  column-backed** — `oneToMany`/`manyToMany`/`meta`/`transient`, plus the
+  `id`/`slug`/`context` system columns on a custom-primary-key class, which the
+  schema generator omits — all of which otherwise reach the driver as
+  `no such column` and surface as a 500. The whitelist is skipped only for
+  manifest-less inline test classes (#869), exactly as `where` skips it. `where`
+  still whitelists those omitted system columns; that is a pre-existing gap of
+  the same family, not closed here because `where: { id }` is on internal
+  hydration paths.
+- Every generated list surface (REST, SvelteKit, MCP, and the emitted stdio MCP
+  runtime) pages with `ORDER BY created_at DESC, <pk> ASC` unless the caller
+  supplies `orderBy` — `LIMIT`/`OFFSET` with no ordering is not pagination, and
+  `created_at` alone still ties. `<pk>` follows a declared
+  `@field({ primaryKey: true })` (read from `_meta.primaryKey` in manifests)
+  because custom-primary-key classes have no synthetic `id` column. The stdio
+  runtime gets the per-object ordering baked in via `RuntimeOptions.listOrderBy`;
+  it cannot resolve a primary key on its own. Index: #2363.
+- `listByIds()` chunks its `IN` list at `IN_LIST_CHUNK_SIZE` (900), like the
+  relationship/junction/hierarchy loaders.
+
+Keyset pagination is deliberately out of scope.