json-rest-api 2.0.0 → 2.2.0

package/docs/API.md

@@ -78,6 +78,7 @@ a separate context for each enlisted operation. See [transaction support](#trans
    - [PUT - Replace Resource](#put---replace-resource)
    - [PATCH - Update Resource](#patch---update-resource)
    - [DELETE - Remove Resource](#delete---remove-resource)
+   - [AGGREGATE - Grouped Totals](#aggregate---grouped-totals)
 3. [Relationship Methods](#relationship-methods)
    - [getRelated - Retrieve Related Resources](#getrelated---retrieve-related-resources)
    - [getRelationship - Retrieve Relationship Identifiers](#getrelationship---retrieve-relationship-identifiers)
@@ -1136,6 +1137,68 @@ savepoints cannot own library writes.
 
 ---
 
+### AGGREGATE - Grouped Totals
+
+Groups the records a collection query would select and computes measures for
+each group, on ordinary and AnyAPI storage alike.
+
+#### Method Signature
+
+```javascript
+await api.resources.sales.aggregate(params, context)
+```
+
+#### Parameters
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `groupBy` | string[] | Fields to group by. Omit for one row over all selected records. |
+| `measures` | object | Output names mapped to `{ count \| countDistinct \| sum \| avg \| min \| max: field }`. Only `count` accepts `'*'`. |
+| `queryParams.filters` | object | Search schema filters, validated as for `query`. |
+| `queryParams.sort` | string[] | Grouped field or measure names, `-` for descending. Grouped fields break ties. |
+| `queryParams.page` | object | `size` and `number`, for grouped results only. `size` defaults to `queryDefaultLimit` and cannot exceed `queryMaxLimit`. |
+| `transaction` | Knex transaction | Optional transaction to read in. |
+
+`sum` and `avg` take numeric fields; `min` and `max` also take strings, dates and
+times. Grouping and counting accept any scalar stored field, including belongs-to
+foreign keys. Fields that are `hidden`, virtual, computed, or that have a getter or
+`storage.serialize` are rejected, because SQL would group or sum values other than
+the ones clients read. `normallyHidden` fields are allowed.
+
+#### Return Value
+
+```javascript
+{
+  data: [{ salesperson: 'Ann', total: 150.5, count: 2 }],
+  meta: { page: { size: 20, number: 1, hasMore: false } } // grouped results only
+}
+```
+
+Both formats return these plain rows. Group values and `min`/`max` use the
+field's public representation; counts are numbers, and `sum`/`avg` are numbers
+except on decimal fields (`precision` and `scale`), which keep the driver's value.
+NULL sorts as the lowest value on every database.
+
+#### Security
+
+`aggregate` checks permissions as a `query` (`method: 'query'`, with
+`originalContext.method === 'aggregate'`). Row policies, autofilters and other
+`knexQueryFiltering` hooks run with `queryPurpose: 'aggregate'`. The aggregate reads
+only the IDs the filtered query selects, so joins added by filters or policies
+never count a record twice.
+
+#### Examples
+
+```javascript
+const totals = await api.resources.sales.aggregate({
+  groupBy: ['salesperson'],
+  measures: { total: { sum: 'amount' }, count: { count: '*' }, avg: { avg: 'amount' } },
+  queryParams: { filters: { closed: true }, sort: ['-total'], page: { size: 10 } }
+}, { subject })
+```
+
+---
+
 ## Relationship Methods
 
 ### getRelated - Retrieve Related Resources
@@ -2039,6 +2102,23 @@ database writes; it is not a persisted UI-state field. Use a setter when a
 virtual input needs to prepare other stored attributes. A `virtualFields` array
 is not the resource declaration contract. See [virtual-field behavior](GUIDE/12-field-transformations.md).
 
+#### Email and UUID Fields
+
+`type: 'email'` and `type: 'uuid'` store trimmed strings and reject values that
+are not an email address or a UUID. Filters with a text operator (`like`,
+`contains`, `startsWith`, `endsWith`) accept fragments. Registration rejects any
+type json-rest-schema cannot cast, so a misspelt type fails at `addResource`
+rather than on the first write.
+
+#### Retired Fields
+
+`retired: true` keeps a stored field readable, filterable and sortable while
+rejecting every value written to it, including `null`, with the `retired_field`
+rule. `required` and `defaultTo` no longer apply, and PUT leaves its stored value
+alone. Only stored attributes can be retired, not belongs-to fields. AnyAPI
+resources can retire fields at run time with
+[`retireKnexFields`](GUIDE/21-schema-and-migrations.md#retiring-and-migrating-anyapi-fields).
+
 #### Field Transformations
 
 Getters and setters receive a value and an injected context. Computed fields

package/docs/GUIDE/17-row-policies.md

@@ -286,6 +286,7 @@ Policies are reused for the resource selections that define externally visible d
 | --- | --- |
 | `collection` | Top-level collection query, before sorting and pagination. On AnyAPI this filtered builder is also cloned for the matching count. |
 | `count` | A separate count query, including native offset-pagination counts and standalone count helpers. |
+| `aggregate` | The records a resource `aggregate()` call groups and measures. |
 | `single` | Minimal lookup used by `get`, and by existing-record preflights for `put`, `patch`, and `delete`. |
 | `include` | A target resource loaded into JSON:API `included` data. |
 | `search-join` | A related resource used by a cross-resource or polymorphic search, reference filter, or reference sort, including intermediate search path segments. |

package/docs/GUIDE/21-schema-and-migrations.md

@@ -297,7 +297,8 @@ This optional map may cover a subset of the new stored fields; remaining fields
 use the existing allocator. An initial resource map must still cover all stored
 fields. Invalid, occupied or unknown slots and existing field names reject the
 addition. Computed and virtual fields need no slots and cannot have overrides.
-AnyAPI `alterKnexFields` remains explicitly unsupported.
+AnyAPI `alterKnexFields` remains explicitly unsupported; change a field's type with
+[`migrateField`](#retiring-and-migrating-anyapi-fields).
 
 One call uses one owned metadata transaction. Allocation failure rolls back the
 batch and leaves the runtime schema unchanged. Added getters, setters, defaults,
@@ -307,26 +308,165 @@ separate DDL behavior.
 
 Existing rows are not backfilled: unused slots stay null, and defaults apply to
 subsequent writes. Function defaults do not run during allocation. Backfill old
-rows explicitly when needed. Declare the expanded schema and complete slot map
-in app source for restart, including callbacks that persisted JSON cannot store.
+rows explicitly when needed.
+
+Additions are stored with their search schema, so they survive a restart
+without being declared again. Persisted JSON cannot hold callbacks: getters,
+setters, function defaults, serializers and `compute` functions exist only in the
+process that added them. A server that loads such a field without declaring it
+in code hides the field and logs a warning instead of serving it without its
+callbacks. Declare fields with callbacks in app source on every server; a
+declared field always takes precedence over a stored addition of the same name.
 
 Explicit maps now take effect during registration. Repeating `createKnexTable()`
 ensures canonical tables and refreshes schema information without replacing
 field metadata or removing additions.
 
-Canonical registration allocates slots from the enriched compiled definitions;
-it does not compile a second schema from persisted JSON. Schema/search hooks run
-once per registration or `addKnexFields` compilation. Repeated table creation
-does not rerun them. Search hook changes therefore survive descriptor refresh.
-
-Re-registration rejects removing or remapping existing stored fields while the
-resource has records. Preserve a complete `canonicalFieldsMap` when declaration
-order changes, and perform an explicit data migration for layout changes. See
-the [schema enrichment migration steps](33-migrating-to-v2.md#schema-enrichment-and-canonical-registration),
+Canonical registration allocates slots from the enriched compiled definitions.
+Stored additions that the declaration does not repeat are compiled from their
+persisted JSON. Schema/search hooks run once per registration, `addKnexFields`
+compilation or reload that changes the field set. Repeated table creation does
+not rerun them. Search hook changes therefore survive descriptor refresh.
+
+Registration never moves a stored field: existing fields keep their slots
+whatever the declaration order, and new fields take the first free slot. A field
+the declaration omits keeps its mapping while the resource has records, because
+servers still running older code may use it; this declaration stops publishing
+it. Unpopulated omitted fields are released. Changing a populated field's storage
+type is still rejected and needs an explicit data migration. See the
+[schema enrichment migration steps](33-migrating-to-v2.md#schema-enrichment-and-canonical-registration),
 including the consequences for hooks that were previously overwritten.
 
-Deep `AnyapiRegistry.allocateField` calls persist the field definition and its
-allocation together. Borrowed transactions own their reads/writes and bypass the
+Registering an unchanged declaration writes nothing. Concurrent schema changes
+to one resource queue on its metadata row, and a unique index stops two fields
+from sharing a slot. Each committed change increments the resource's
+`schema_version`. On MySQL, the registry runs its own transactions at READ
+COMMITTED, because REPEATABLE READ makes concurrent creation of different
+resources or tenants deadlock. A transaction you pass in keeps its own isolation
+level.
+
+### AnyAPI across several servers
+
+Each server compiles resources into memory. When one server adds fields, the
+others keep their published schema until they reload it:
+
+```js
+await api.use(RestApiAnyapiKnexPlugin, {
+  knex,
+  tenantId,
+  // Runs after each committed change; publish it to the other servers.
+  onSchemaChanged: change => notifyServers(change),
+  // Safety net for missed notifications.
+  schemaCheckInterval: 5000
+})
+
+// On a server that receives a notification:
+await api.anyapi.handleSchemaChange(change)
+```
+
+`handleSchemaChange({ tenant, resource, version })` reloads the resource only when
+the version is newer than the published one, and returns whether it did.
+`api.anyapi.getSchemaVersion('items')` reads the stored version with one query.
+`api.resources.items.reloadSchema()` reloads unconditionally and resolves to the
+published version. Reloads are read-only: they combine the stored definition
+with this server's declaration and never re-register it.
+
+With `schemaCheckInterval`, request methods compare the stored version at most
+once per interval; `0` checks on every request. A request rejected by validation
+also checks the version once and, if the resource changed, reloads and retries.
+A write whose transaction may have committed is never retried. Calls given a
+caller transaction skip both checks, because that transaction may hold
+uncommitted metadata. A rejected `onSchemaChanged` callback is logged; the
+committed change stands.
+
+A server can serve resources it never declared. `await api.anyapi.loadResource('items')`
+loads a stored resource, with the resources it relates to, and resolves to it, or to
+`null` when none is stored. `api.anyapi.loadResources()` loads every stored resource
+of the tenant. A stored resource whose non-field options needed callbacks, such as
+`normalizeId`, must be declared in code. To load resources on demand over HTTP,
+call `loadResource` from middleware placed before the API router.
+
+### Retiring and migrating AnyAPI fields
+
+Slots are never released while records exist, so AnyAPI retires fields instead of
+dropping them:
+
+```js
+await api.resources.contacts.retireKnexFields(['phone'])
+```
+
+A retired field keeps its slot and its values, stays readable, filterable and
+sortable, and rejects writes as a [`retired: true`](../API.md#retired-fields) field
+does. The mark is stored with the resource definition, so a server that registers
+an older declaration still listing the field keeps it retired. Its name stays taken.
+
+`migrateField` changes a field's type by copying its values into a new field:
+
+```js
+await api.resources.contacts.migrateField({
+  from: 'phone',
+  to: { name: 'phoneNumber', type: 'number', nullable: true, search: true },
+  transform: (value, { id }) => value === null ? null : Number(value.replace(/\D/g, ''))
+})
+```
+
+In one transaction it allocates a slot for the new field, converts every record's
+value, and retires the old field. Each converted value must pass the new field's
+validation; without `transform`, that validation casts the old value directly. A
+value it rejects names the record and rolls everything back. The new field must be
+a plain stored attribute: add fields with callbacks with `addKnexFields` and
+backfill them. Values are written directly, so setters, hooks and record versions
+do not run. Records are converted one by one, which suits per-tenant tables
+better than very large ones. Other servers keep accepting writes to the old field
+until the migration commits, and those writes do not reach the new field, so run
+it while nothing writes the field.
+
+Both methods bump `schema_version` and call `onSchemaChanged` once after commit.
+`AnyapiRegistry.retireFields({ tenant, resource, fields }, { transaction })` is the
+deep form of retirement.
+
+### AnyAPI tenant instances
+
+One API instance per tenant stays cheap when the instances share their start-up work:
+
+```js
+// In a migration:
+await ensureAnyApiSchema(knex, { slots: { string: 60, number: 30, boolean: 20, date: 20, json: 10, belongsTo: 15 } })
+
+// In the application:
+const registry = new AnyapiRegistry({ knex, cacheSize: 5000 })
+const tenantApi = async tenantId => {
+  const api = new JsonRestApi()
+  await api.use(RestApiPlugin)
+  await api.use(RestApiAnyapiKnexPlugin, { knex, tenantId, registry, ensureSchema: false, ownsKnex: false })
+  await api.anyapi.loadResources()
+  return api
+}
+```
+
+A shared registry reads database capabilities and the slot layout once, and holds
+one descriptor cache, bounded by `cacheSize` (default 100). `ensureSchema: false`
+skips table checks and the logical ID backfill at start-up and in `createKnexTable()`;
+migrations must then run `ensureAnyApiSchema`. `ownsKnex: false` keeps `api.release()`
+from destroying the shared Knex instance. A dropped instance needs no other cleanup.
+
+### AnyAPI slot limits
+
+Slot limits belong to the database: every server reads the slot columns that
+`any_records` actually has. The defaults are 10 string, 10 number, 5 boolean, 5
+date, 5 JSON and 5 belongsTo slots. `ensureAnyApiSchema(knex, { slots })`, or the
+plugin's `slots` option when it ensures the schema, adds missing columns. Lower
+counts never drop columns. A server whose pool is exhausted rereads the layout
+once, so a migration that adds slots takes effect without a restart.
+
+On PostgreSQL, adding a nullable column without a default does not rewrite the
+table. MySQL can add columns instantly from 8.0.29; earlier versions may copy the
+table. MySQL limits a row to 65,535 bytes and each belongsTo slot adds two
+string columns, so check the row size before raising that pool substantially.
+
+Deep `AnyapiRegistry.allocateField` and `allocateFields` calls persist the field
+definitions and their allocation together. `allocateFields` adds a batch under one
+version; pass each field's uncompiled `declaration` to store it for other servers. Borrowed transactions own their reads/writes and bypass the
 global descriptor cache; the registry does not commit, roll back or publish their
 uncommitted descriptors globally. After commit, call
 `invalidateDescriptor(tenant, resource)` before an ordinary cached read. Deep

package/docs/GUIDE/30-backend-capabilities.md

@@ -62,11 +62,14 @@ changes can be omitted with warnings, and destructive `down` migrations are not
 generated automatically. SQLite check/enum changes can require a reviewed table
 rebuild. Native SET fields require an explicit MySQL target.
 
-AnyAPI field additions persist slot metadata; `alterKnexFields` is unsupported.
-Re-registering populated resources cannot remove or remap stored fields. Preserve
-slot maps and migrate stored data explicitly when changing layouts. Old canonical
-temporal storage needs the documented data migration, not just a package update.
-See [table helpers and migration limits](21-schema-and-migrations.md).
+AnyAPI field additions persist slot metadata and their definitions;
+`alterKnexFields` is unsupported. Registration keeps stored fields in their slots
+and keeps the mapping of populated fields a declaration omits; changing a populated
+field's storage type needs an explicit data migration. Servers sharing a database
+see each other's additions after a reload, notification or schema check, and slot
+limits follow the columns the database has. Old canonical temporal storage needs
+the documented data migration, not just a package update. See
+[table helpers and migration limits](21-schema-and-migrations.md#anyapi-across-several-servers).
 
 ## Transaction ownership and failures
 

package/docs/GUIDE/33-migrating-to-v2.md

@@ -1038,6 +1038,15 @@ Use `HookHandler<Event, AppContext>` for known hooks and
 `RuntimeHook<PluginContext>` for arbitrary plugin events; see the
 [hook guide](13-hooks-and-lifecycle.md).
 
+## Field type names
+
+`addResource` now rejects a field whose type json-rest-schema cannot cast, naming
+the field and type. Such fields used to register and then fail on their first
+write with `No casting function for type`. AnyAPI no longer lists `text`, `float`,
+`decimal`, `datetime` or `json`, which no storage could write; use `string`,
+`number`, `dateTime` and `object` or `array`. `email` and `uuid` are now real
+string types with format checks.
+
 ## Object and array attributes
 
 Declare structured fields using their public JSON type:
@@ -1466,6 +1475,16 @@ Missing `fields`, duplicate fields and invalid maps now throw instead of silentl
 succeeding or leaving unusable metadata. See
 [AnyAPI field additions](21-schema-and-migrations.md#anyapi-field-additions).
 
+`ensureAnyApiSchema` now adds `schema_version` and `definition_json` to
+`any_resource_configs` and a unique slot index to `any_field_configs`. It refuses
+to add the index while two fields of one resource share a slot, naming the slot;
+reassign one of them first. Deployments using `ensureSchema: false` must run
+`ensureAnyApiSchema` in a migration. Registration no longer rejects a declaration
+that omits a populated field or reorders fields: stored slots stay where they
+are, and omitted populated fields keep their data without being published.
+Fields added before this release that the declaration does not repeat are kept
+and published from their stored definitions.
+
 Explicit maps take effect during registration, before `createKnexTable()`.
 Repeating that method no longer re-registers the initial schema and discards
 additions. Keep the expanded declaration and complete map in app source for
@@ -2805,13 +2824,14 @@ compilation. Canonical metadata no longer invents an attribute declaration for
 an implicit resource ID; metadata consumers should use `schemaInfo.idProperty`
 to identify the resource ID configuration.
 
-Enrichment runs once at resource registration, and once for each canonical
-`addKnexFields` compilation. Repeated canonical `createKnexTable()` refreshes
-storage metadata without rerunning enrichment or restoring removed search
-fields. Deep registry allocation does not update a running resource's compiled
-schema; use the resource helper and retain its expanded declaration in source.
-Declare callbacks in source again after restart: persisted JSON cannot restore
-functions. Ordinary `addKnexFields` retains its DDL-only behavior.
+Enrichment runs once at resource registration, once for each canonical
+`addKnexFields` compilation, and once for a reload that changes the field set.
+Repeated canonical `createKnexTable()` refreshes storage metadata without
+rerunning enrichment or restoring removed search fields. Deep registry allocation
+does not update a running resource's compiled schema; use the resource helper or
+`reloadSchema()`. Additions survive restarts without being declared again, but
+persisted JSON cannot restore functions: declare fields with callbacks in source,
+or servers hide them. Ordinary `addKnexFields` retains its DDL-only behavior.
 
 Fastify now resolves the current cached request contract when validating a
 payload. A belongs-to alias added through canonical `addKnexFields` can be used

package/index.d.ts

@@ -6,6 +6,7 @@ export type * from './types/relationship-methods.js'
 export type * from './types/bulk-methods.js'
 export type { ManagedTransaction, ManagedTransactionParam, TransactionMethods } from './types/transactions.js'
 export type * from './types/plugin-options.js'
+export * from './types/anyapi.js'
 export type * from './types/file-detectors.js'
 export type * from './types/fastify-options.js'
 export type * from './types/row-policy.js'

package/index.js

@@ -13,6 +13,8 @@ export { SocketIOPlugin } from './plugins/core/socketio-plugin.js'
 // Database plugins
 export { RestApiKnexPlugin } from './plugins/core/rest-api-knex-plugin.js'
 export { RestApiAnyapiKnexPlugin } from './plugins/core/rest-api-anyapi-knex-plugin.js'
+export { AnyapiRegistry } from './plugins/core/lib/anyapi/anyapi-registry.js'
+export { ensureAnyApiSchema } from './plugins/core/lib/anyapi/schema-utils.js'
 
 // Connector plugins
 export { ExpressPlugin } from './plugins/core/connectors/express-plugin.js'

package/lib/knex-transaction.js

@@ -1,12 +1,16 @@
 // @ts-check
 /** @import { OwnedTransaction, TransactionOutcome } from './transaction-types.js' */
 /** @import { StorageDatabase } from '../plugins/core/lib/storage/storage-types.js' */
-/** @typedef {{ cleanupErrors?: Array<{ phase: string, error: unknown }>, [serializableTransaction]?: boolean }} CleanupContext */
+/** @typedef {{ cleanupErrors?: Array<{ phase: string, error: unknown }>, [serializableTransaction]?: boolean, [readCommittedTransaction]?: boolean }} CleanupContext */
 /** @type {WeakMap<OwnedTransaction, (error?: unknown) => Promise<void>>} */
 const transactionConnections = new WeakMap()
 
 // Internal connector request; HTTP input cannot select transaction isolation.
 export const serializableTransaction = Symbol('serializableTransaction')
+// MySQL's default REPEATABLE READ gap-locks locking reads of missing rows, so
+// concurrent creators of different rows deadlock on insert. PostgreSQL and
+// SQLite already behave as requested.
+export const readCommittedTransaction = Symbol('readCommittedTransaction')
 
 // Own the lease without changing the driver's SQL-completion promise.
 /**
@@ -30,9 +34,11 @@ export async function createKnexTransaction (knex, context = {}) {
     return releasePromise
   }
   try {
-    const serializable = context[serializableTransaction] === true
-    const sqlite = ['sqlite3', 'better-sqlite3'].includes(knex.client.config.client)
-    transaction = await knex.transaction({ connection, ...(serializable && !sqlite ? { isolationLevel: 'serializable' } : {}) })
+    const client = knex.client.config.client
+    const isolationLevel = context[serializableTransaction] === true
+      ? !['sqlite3', 'better-sqlite3'].includes(client) && 'serializable'
+      : context[readCommittedTransaction] === true && ['mysql', 'mysql2'].includes(client) && 'read committed'
+    transaction = await knex.transaction({ connection, ...(isolationLevel ? { isolationLevel } : {}) })
     transactionConnections.set(transaction, release)
     transaction.executionPromise.then(
       () => release(),

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "json-rest-api",
-  "version": "2.0.0",
+  "version": "2.2.0",
   "main": "index.js",
   "types": "index.d.ts",
   "type": "module",