json-rest-api 2.0.0 → 2.1.0

package/docs/GUIDE/21-schema-and-migrations.md

@@ -307,26 +307,125 @@ separate DDL behavior.
 
 Existing rows are not backfilled: unused slots stay null, and defaults apply to
 subsequent writes. Function defaults do not run during allocation. Backfill old
-rows explicitly when needed. Declare the expanded schema and complete slot map
-in app source for restart, including callbacks that persisted JSON cannot store.
+rows explicitly when needed.
+
+Additions are stored with their search schema, so they survive a restart
+without being declared again. Persisted JSON cannot hold callbacks: getters,
+setters, function defaults, serializers and `compute` functions exist only in the
+process that added them. A server that loads such a field without declaring it
+in code hides the field and logs a warning instead of serving it without its
+callbacks. Declare fields with callbacks in app source on every server; a
+declared field always takes precedence over a stored addition of the same name.
 
 Explicit maps now take effect during registration. Repeating `createKnexTable()`
 ensures canonical tables and refreshes schema information without replacing
 field metadata or removing additions.
 
-Canonical registration allocates slots from the enriched compiled definitions;
-it does not compile a second schema from persisted JSON. Schema/search hooks run
-once per registration or `addKnexFields` compilation. Repeated table creation
-does not rerun them. Search hook changes therefore survive descriptor refresh.
-
-Re-registration rejects removing or remapping existing stored fields while the
-resource has records. Preserve a complete `canonicalFieldsMap` when declaration
-order changes, and perform an explicit data migration for layout changes. See
-the [schema enrichment migration steps](33-migrating-to-v2.md#schema-enrichment-and-canonical-registration),
+Canonical registration allocates slots from the enriched compiled definitions.
+Stored additions that the declaration does not repeat are compiled from their
+persisted JSON. Schema/search hooks run once per registration, `addKnexFields`
+compilation or reload that changes the field set. Repeated table creation does
+not rerun them. Search hook changes therefore survive descriptor refresh.
+
+Registration never moves a stored field: existing fields keep their slots
+whatever the declaration order, and new fields take the first free slot. A field
+the declaration omits keeps its mapping while the resource has records, because
+servers still running older code may use it; this declaration stops publishing
+it. Unpopulated omitted fields are released. Changing a populated field's storage
+type is still rejected and needs an explicit data migration. See the
+[schema enrichment migration steps](33-migrating-to-v2.md#schema-enrichment-and-canonical-registration),
 including the consequences for hooks that were previously overwritten.
 
-Deep `AnyapiRegistry.allocateField` calls persist the field definition and its
-allocation together. Borrowed transactions own their reads/writes and bypass the
+Registering an unchanged declaration writes nothing. Concurrent schema changes
+to one resource queue on its metadata row, and a unique index stops two fields
+from sharing a slot. Each committed change increments the resource's
+`schema_version`. On MySQL, the registry runs its own transactions at READ
+COMMITTED, because REPEATABLE READ makes concurrent creation of different
+resources or tenants deadlock. A transaction you pass in keeps its own isolation
+level.
+
+### AnyAPI across several servers
+
+Each server compiles resources into memory. When one server adds fields, the
+others keep their published schema until they reload it:
+
+```js
+await api.use(RestApiAnyapiKnexPlugin, {
+  knex,
+  tenantId,
+  // Runs after each committed change; publish it to the other servers.
+  onSchemaChanged: change => notifyServers(change),
+  // Safety net for missed notifications.
+  schemaCheckInterval: 5000
+})
+
+// On a server that receives a notification:
+await api.anyapi.handleSchemaChange(change)
+```
+
+`handleSchemaChange({ tenant, resource, version })` reloads the resource only when
+the version is newer than the published one, and returns whether it did.
+`api.resources.items.reloadSchema()` reloads unconditionally and resolves to the
+published version. Reloads are read-only: they combine the stored definition
+with this server's declaration and never re-register it.
+
+With `schemaCheckInterval`, request methods compare the stored version at most
+once per interval; `0` checks on every request. A request rejected by validation
+also checks the version once and, if the resource changed, reloads and retries.
+A write whose transaction may have committed is never retried. Calls given a
+caller transaction skip both checks, because that transaction may hold
+uncommitted metadata. A rejected `onSchemaChanged` callback is logged; the
+committed change stands.
+
+A server can serve resources it never declared. `await api.anyapi.loadResource('items')`
+loads a stored resource, with the resources it relates to, and resolves to it, or to
+`null` when none is stored. `api.anyapi.loadResources()` loads every stored resource
+of the tenant. A stored resource whose non-field options needed callbacks, such as
+`normalizeId`, must be declared in code. To load resources on demand over HTTP,
+call `loadResource` from middleware placed before the API router.
+
+### AnyAPI tenant instances
+
+One API instance per tenant stays cheap when the instances share their start-up work:
+
+```js
+// In a migration:
+await ensureAnyApiSchema(knex, { slots: { string: 60, number: 30, boolean: 20, date: 20, json: 10, belongsTo: 15 } })
+
+// In the application:
+const registry = new AnyapiRegistry({ knex, cacheSize: 5000 })
+const tenantApi = async tenantId => {
+  const api = new JsonRestApi()
+  await api.use(RestApiPlugin)
+  await api.use(RestApiAnyapiKnexPlugin, { knex, tenantId, registry, ensureSchema: false, ownsKnex: false })
+  await api.anyapi.loadResources()
+  return api
+}
+```
+
+A shared registry reads database capabilities and the slot layout once, and holds
+one descriptor cache, bounded by `cacheSize` (default 100). `ensureSchema: false`
+skips table checks and the logical ID backfill at start-up and in `createKnexTable()`;
+migrations must then run `ensureAnyApiSchema`. `ownsKnex: false` keeps `api.release()`
+from destroying the shared Knex instance. A dropped instance needs no other cleanup.
+
+### AnyAPI slot limits
+
+Slot limits belong to the database: every server reads the slot columns that
+`any_records` actually has. The defaults are 10 string, 10 number, 5 boolean, 5
+date, 5 JSON and 5 belongsTo slots. `ensureAnyApiSchema(knex, { slots })`, or the
+plugin's `slots` option when it ensures the schema, adds missing columns. Lower
+counts never drop columns. A server whose pool is exhausted rereads the layout
+once, so a migration that adds slots takes effect without a restart.
+
+On PostgreSQL, adding a nullable column without a default does not rewrite the
+table. MySQL can add columns instantly from 8.0.29; earlier versions may copy the
+table. MySQL limits a row to 65,535 bytes and each belongsTo slot adds two
+string columns, so check the row size before raising that pool substantially.
+
+Deep `AnyapiRegistry.allocateField` and `allocateFields` calls persist the field
+definitions and their allocation together. `allocateFields` adds a batch under one
+version; pass each field's uncompiled `declaration` to store it for other servers. Borrowed transactions own their reads/writes and bypass the
 global descriptor cache; the registry does not commit, roll back or publish their
 uncommitted descriptors globally. After commit, call
 `invalidateDescriptor(tenant, resource)` before an ordinary cached read. Deep

package/docs/GUIDE/30-backend-capabilities.md

@@ -62,11 +62,14 @@ changes can be omitted with warnings, and destructive `down` migrations are not
 generated automatically. SQLite check/enum changes can require a reviewed table
 rebuild. Native SET fields require an explicit MySQL target.
 
-AnyAPI field additions persist slot metadata; `alterKnexFields` is unsupported.
-Re-registering populated resources cannot remove or remap stored fields. Preserve
-slot maps and migrate stored data explicitly when changing layouts. Old canonical
-temporal storage needs the documented data migration, not just a package update.
-See [table helpers and migration limits](21-schema-and-migrations.md).
+AnyAPI field additions persist slot metadata and their definitions;
+`alterKnexFields` is unsupported. Registration keeps stored fields in their slots
+and keeps the mapping of populated fields a declaration omits; changing a populated
+field's storage type needs an explicit data migration. Servers sharing a database
+see each other's additions after a reload, notification or schema check, and slot
+limits follow the columns the database has. Old canonical temporal storage needs
+the documented data migration, not just a package update. See
+[table helpers and migration limits](21-schema-and-migrations.md#anyapi-across-several-servers).
 
 ## Transaction ownership and failures
 

package/docs/GUIDE/33-migrating-to-v2.md

@@ -1466,6 +1466,16 @@ Missing `fields`, duplicate fields and invalid maps now throw instead of silentl
 succeeding or leaving unusable metadata. See
 [AnyAPI field additions](21-schema-and-migrations.md#anyapi-field-additions).
 
+`ensureAnyApiSchema` now adds `schema_version` and `definition_json` to
+`any_resource_configs` and a unique slot index to `any_field_configs`. It refuses
+to add the index while two fields of one resource share a slot, naming the slot;
+reassign one of them first. Deployments using `ensureSchema: false` must run
+`ensureAnyApiSchema` in a migration. Registration no longer rejects a declaration
+that omits a populated field or reorders fields: stored slots stay where they
+are, and omitted populated fields keep their data without being published.
+Fields added before this release that the declaration does not repeat are kept
+and published from their stored definitions.
+
 Explicit maps take effect during registration, before `createKnexTable()`.
 Repeating that method no longer re-registers the initial schema and discards
 additions. Keep the expanded declaration and complete map in app source for
@@ -2805,13 +2815,14 @@ compilation. Canonical metadata no longer invents an attribute declaration for
 an implicit resource ID; metadata consumers should use `schemaInfo.idProperty`
 to identify the resource ID configuration.
 
-Enrichment runs once at resource registration, and once for each canonical
-`addKnexFields` compilation. Repeated canonical `createKnexTable()` refreshes
-storage metadata without rerunning enrichment or restoring removed search
-fields. Deep registry allocation does not update a running resource's compiled
-schema; use the resource helper and retain its expanded declaration in source.
-Declare callbacks in source again after restart: persisted JSON cannot restore
-functions. Ordinary `addKnexFields` retains its DDL-only behavior.
+Enrichment runs once at resource registration, once for each canonical
+`addKnexFields` compilation, and once for a reload that changes the field set.
+Repeated canonical `createKnexTable()` refreshes storage metadata without
+rerunning enrichment or restoring removed search fields. Deep registry allocation
+does not update a running resource's compiled schema; use the resource helper or
+`reloadSchema()`. Additions survive restarts without being declared again, but
+persisted JSON cannot restore functions: declare fields with callbacks in source,
+or servers hide them. Ordinary `addKnexFields` retains its DDL-only behavior.
 
 Fastify now resolves the current cached request contract when validating a
 payload. A belongs-to alias added through canonical `addKnexFields` can be used

package/index.d.ts

@@ -6,6 +6,7 @@ export type * from './types/relationship-methods.js'
 export type * from './types/bulk-methods.js'
 export type { ManagedTransaction, ManagedTransactionParam, TransactionMethods } from './types/transactions.js'
 export type * from './types/plugin-options.js'
+export * from './types/anyapi.js'
 export type * from './types/file-detectors.js'
 export type * from './types/fastify-options.js'
 export type * from './types/row-policy.js'

package/index.js

@@ -13,6 +13,8 @@ export { SocketIOPlugin } from './plugins/core/socketio-plugin.js'
 // Database plugins
 export { RestApiKnexPlugin } from './plugins/core/rest-api-knex-plugin.js'
 export { RestApiAnyapiKnexPlugin } from './plugins/core/rest-api-anyapi-knex-plugin.js'
+export { AnyapiRegistry } from './plugins/core/lib/anyapi/anyapi-registry.js'
+export { ensureAnyApiSchema } from './plugins/core/lib/anyapi/schema-utils.js'
 
 // Connector plugins
 export { ExpressPlugin } from './plugins/core/connectors/express-plugin.js'

package/lib/knex-transaction.js

@@ -1,12 +1,16 @@
 // @ts-check
 /** @import { OwnedTransaction, TransactionOutcome } from './transaction-types.js' */
 /** @import { StorageDatabase } from '../plugins/core/lib/storage/storage-types.js' */
-/** @typedef {{ cleanupErrors?: Array<{ phase: string, error: unknown }>, [serializableTransaction]?: boolean }} CleanupContext */
+/** @typedef {{ cleanupErrors?: Array<{ phase: string, error: unknown }>, [serializableTransaction]?: boolean, [readCommittedTransaction]?: boolean }} CleanupContext */
 /** @type {WeakMap<OwnedTransaction, (error?: unknown) => Promise<void>>} */
 const transactionConnections = new WeakMap()
 
 // Internal connector request; HTTP input cannot select transaction isolation.
 export const serializableTransaction = Symbol('serializableTransaction')
+// MySQL's default REPEATABLE READ gap-locks locking reads of missing rows, so
+// concurrent creators of different rows deadlock on insert. PostgreSQL and
+// SQLite already behave as requested.
+export const readCommittedTransaction = Symbol('readCommittedTransaction')
 
 // Own the lease without changing the driver's SQL-completion promise.
 /**
@@ -30,9 +34,11 @@ export async function createKnexTransaction (knex, context = {}) {
     return releasePromise
   }
   try {
-    const serializable = context[serializableTransaction] === true
-    const sqlite = ['sqlite3', 'better-sqlite3'].includes(knex.client.config.client)
-    transaction = await knex.transaction({ connection, ...(serializable && !sqlite ? { isolationLevel: 'serializable' } : {}) })
+    const client = knex.client.config.client
+    const isolationLevel = context[serializableTransaction] === true
+      ? !['sqlite3', 'better-sqlite3'].includes(client) && 'serializable'
+      : context[readCommittedTransaction] === true && ['mysql', 'mysql2'].includes(client) && 'read committed'
+    transaction = await knex.transaction({ connection, ...(isolationLevel ? { isolationLevel } : {}) })
     transactionConnections.set(transaction, release)
     transaction.executionPromise.then(
       () => release(),

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "json-rest-api",
-  "version": "2.0.0",
+  "version": "2.1.0",
   "main": "index.js",
   "types": "index.d.ts",
   "type": "module",