json-rest-api 1.0.28 → 2.0.0

package/docs/GUIDE/28-socketio.md

@@ -0,0 +1,325 @@
+---
+title: "Socket.IO integration"
+chapter: 28
+chapter_label: "28"
+---
+
+# 28. Socket.IO integration
+
+`SocketIOPlugin` sends `subscription.update` messages after resource changes.
+Messages identify a changed resource; clients fetch records through the resource
+API using their own fields, includes and authorization context.
+
+## Runnable public example
+
+This example deliberately exposes a public countries resource. It uses a local
+in-memory database and accepts anonymous subscriptions.
+
+```js
+import { JsonRestApi } from 'json-rest-api'
+import { RestApiPlugin, RestApiKnexPlugin, ExpressPlugin, SocketIOPlugin } from 'json-rest-api'
+import { createServer } from 'node:http'
+import express from 'express'
+import knex from 'knex'
+
+const db = knex({
+  client: 'better-sqlite3',
+  connection: { filename: ':memory:' },
+  useNullAsDefault: true
+})
+const api = new JsonRestApi({ name: 'socket-example' })
+const app = express()
+await api.use(RestApiPlugin)
+await api.use(RestApiKnexPlugin, { knex: db })
+await api.use(ExpressPlugin, { mountPath: '/api' })
+await api.use(SocketIOPlugin, { subscriptions: { maxPerSocket: 20 } })
+await api.addResource('countries', {
+  schema: { id: { type: 'id' }, name: { type: 'string', required: true, search: true } }
+})
+await api.resources.countries.createKnexTable()
+api.http.express.mount(app)
+const server = createServer(app)
+await api.startSocketServer(server)
+server.listen(3000)
+```
+
+Install `socket.io` in the server application and `socket.io-client` in the
+client. Close the Socket.IO server and database during application shutdown:
+
+```js
+await new Promise(resolve => api.io.close(resolve))
+await db.destroy()
+```
+
+Socket.IO closes the HTTP server passed to `startSocketServer`. Closing that
+server also initiates graceful shutdown of the Redis clients created by the
+plugin and clears `api.io` and its internal server/client references. Database
+ownership remains with the application; ordinary resource calls still work
+after Socket.IO shuts down. Do not separately quit those plugin-created Redis
+clients after closing Socket.IO.
+
+Only one Socket.IO startup may be active per API instance. Overlapping or
+duplicate starts reject instead of replacing the active server. With Redis
+enabled, both connections must succeed before Socket.IO attaches to the HTTP
+server. Failed Redis startup closes partial connections, preserves the caller's
+HTTP listeners and authentication configuration, and permits a subsequent retry.
+Redis's configured reconnection policy is retained; set
+`redis.socket.reconnectStrategy: false` when startup should fail on its first
+connection error rather than retry.
+
+The Socket.IO path defaults to the HTTP connector mount path plus `/socket.io`,
+or `/socket.io` without a connector. Override it with
+`startSocketServer(server, { path: '/events/socket.io' })`. HTTP resource routes
+continue to use their configured API prefix.
+
+## Subscribe and fetch
+
+After the connection is established:
+
+```js
+import { io } from 'socket.io-client'
+
+const socket = io('http://localhost:3000', { path: '/api/socket.io' })
+socket.on('connect', async () => {
+  const result = await socket.timeout(5000).emitWithAck('subscribe', {
+    resource: 'countries',
+    subscriptionId: 'country-list',
+    filters: { name: 'Australia' }
+  })
+  if (result.error) console.error(result.error)
+})
+socket.on('subscription.update', change => {
+  // Invalidate the affected view, then fetch through your normal API client.
+  console.log(change.resource, change.id, change.action)
+})
+```
+
+Subscription input accepts these fields:
+
+| Field | Meaning |
+| --- | --- |
+| `resource` | Required declared resource name |
+| `subscriptionId` | Optional nonempty string; omitted IDs are generated UUIDs |
+| `filters` | Optional object validated with the resource's search schema |
+
+An acknowledgement is either `{ success: true, data }` or `{ error: { code,
+message } }`. Successful data contains `subscriptionId`, `resource`, normalized
+`filters` and `status: 'active'`. Without an acknowledgement callback, subscribe
+emits `subscription.created` or `subscription.error` instead.
+
+`include` and `fields` are not subscription options: notifications never return
+full records. Keep those options in the client's subsequent GET/query. Unknown
+options receive `UNSUPPORTED_OPTION`; malformed filter containers receive
+`INVALID_FILTERS` instead of becoming unfiltered subscriptions.
+
+IDs must be unique among the socket's active subscriptions. Reusing one returns
+`SUBSCRIPTION_EXISTS`; unsubscribe before replacing it. The default limit is
+100 active subscriptions per socket, controlled by `subscriptions.maxPerSocket`.
+Concurrent async admission hooks cannot exceed the configured limit.
+
+To remove a subscription:
+
+```js
+await socket.timeout(5000).emitWithAck('unsubscribe', { subscriptionId: 'country-list' })
+```
+
+The connection stays subscribed to a resource while it has other subscriptions
+for that resource. Every matching subscription receives its own update, with its
+own subscription ID.
+
+Unsubscribe removes the logical subscription immediately and waits for any
+required room departure before acknowledging success. Room joins and departures
+are ordered per socket, so a concurrent new subscription retains its membership.
+If the adapter rejects departure, the acknowledgement contains `UNSUBSCRIBE_ERROR`;
+the logical subscription remains removed and cannot receive notifications. The
+failure does not imply that adapter room cleanup succeeded. A later subscription
+can join normally; disconnect also ends the socket's room membership.
+
+After reconnecting, submit the desired subscriptions again:
+
+```js
+const result = await socket.timeout(5000).emitWithAck('restore-subscriptions', {
+  subscriptions: [{ resource: 'countries', subscriptionId: 'country-list' }]
+})
+// result.restored contains successful IDs; result.failed contains per-entry errors.
+```
+
+Restoration uses the same validation, permission, uniqueness and limit checks as
+subscribe. It does not replay changes missed during disconnection.
+
+## Notification shape
+
+```json
+{
+  "type": "resource.updated",
+  "resource": "countries",
+  "id": "42",
+  "action": "patch",
+  "subscriptionId": "country-list",
+  "meta": { "timestamp": "2026-09-09T00:00:00.000Z" }
+}
+```
+
+| Change | `type` | `action` |
+| --- | --- | --- |
+| POST or PUT-create | `resource.created` | `post` or `put` |
+| PATCH or PUT replacement | `resource.updated` | `patch` or `put` |
+| Relationship POST/PATCH/DELETE | `resource.updated` | `patch` |
+| Resource DELETE | `resource.deleted` | `delete` |
+
+IDs are strings. Deletion also includes `deletedRecord: { id }`, without deleted
+attributes. Reverse relationship changes notify the parent and the changed
+child resources. A relationship PATCH's nested resource PATCH provides its
+parent update; it is not emitted a second time by the outer relationship hook.
+
+Eligibility uses the resource's ordinary SQL query before and after each write,
+inside that write's transaction. The changed ID is constrained internally; it
+need not be a public search field, and a client's ID filter cannot replace it.
+A record entering or leaving the query result invalidates that subscription.
+Deletion uses the before-write result. Custom `applyFilter(query, value)` search
+fields work directly; remove any Socket.IO-only `filterRecord` callback. There
+is no separate JavaScript predicate to maintain.
+
+These queries run normal query permissions, hooks, filters, row policies and
+autofilters. They use subscriber context, never the writer's context. Query
+failures are logged and count as nonmatches; a successful match before or after
+is sufficient. Applications should account for these additional queries and
+ensure query hooks can run inside the write transaction.
+
+## Authentication and permissions
+
+For an application with authenticated users, install the plugin with a
+server-owned authentication function:
+
+```js
+await api.use(SocketIOPlugin, {
+  auth: {
+    requireAuth: true,
+    authenticate: async ({ socket }) => {
+      const identity = await verifyToken(socket.handshake.auth?.token)
+      return { userId: identity.id, roles: identity.roles, workspaceId: identity.workspaceId, groups: identity.groups }
+    }
+  }
+})
+```
+
+`verifyToken` is application code. A failed authentication rejects the connection.
+Handshake fields do not become trusted identity unless the application explicitly
+enables `allowClientProvidedAuth`; the default is false. `anonymousContext` can
+supply a deliberate public context. `onAuthenticationFailed` can observe a
+failure; an error from that observer does not authorize the connection.
+
+Resource query permission is checked when subscribing and again before each
+notification. A failing notification permission check suppresses that message
+and is logged. It does not undo the committed write. Authentication is performed
+when connecting, so applications that revoke sessions must also invalidate or
+disconnect their active sockets as appropriate.
+
+A `subscriptionFilters` hook can add filters and set trusted query context.
+Populate the same application context your row policies and autofilters need.
+Hook arguments use the resource runtime envelope:
+
+```js
+await api.customize({ hooks: {
+  subscriptionFilters: {
+    functionName: 'limit-subscriptions',
+    handler: ({ context }) => {
+      const { subscription, auth } = context
+      subscription.context = {
+        scopeValues: { workspaceId: auth.workspaceId },
+        visibility: { groups: auth.groups }
+      }
+    }
+  }
+} })
+```
+
+The context property defaults to `{}` and must be an object. It is supplied to
+both admission and notification query permission checks; `auth` always comes
+from the stored subscription identity. Use JSON-serializable context for Redis
+adapter transport. Clients cannot supply `context` in subscribe/restore messages.
+The resulting filters are validated again, and the hook cannot change the
+resource or subscription ID. Replacing a subscription with the same public ID
+creates a new generation: it receives subsequent changes only, not messages
+queued for the removed subscription.
+
+Admission failures are logged with `operation: 'subscribe'`, `phase: 'admission'`
+and the resolved `scopeName`, including individual failures while restoring
+subscriptions. After resolving a resource, the diagnostic formatter redacts
+structured fields marked `hidden` or `normallyHidden` in that resource's schema.
+Unknown resources have no resource-specific field policy. Diagnostic formatting
+does not mutate the original error, and a failed log sink does not replace the
+subscription rejection. This does not redact secrets embedded in arbitrary
+error-message text; avoid putting secret values in messages.
+
+## Transactions, Redis and verification limits
+
+For individual resource and relationship operations, notifications wait for
+commit of the library-owned transaction. Writes that roll back,
+including finish-hook failures after events were queued, emit no successful
+change. A rejected call after commit can still have emitted notifications. These paths are tested on both SQLite storage modes over WebSocket and
+HTTP polling, including relationship changes and child rollback.
+
+Library writes now require the handle from `api.transaction` when joining an
+outer unit. Resource and atomic-bulk child completion hooks run after confirmed
+owner completion; non-atomic bulk retains one outcome per entry. Rolled-back
+units do not flush queued notifications. Explicit callbacks are tested with
+repeated changes to the same record, callback/caught-write/caught-SQL failures,
+and rejected completion hooks on both storage modes and all three databases.
+Direct raw transaction completion is no longer a supported library-write
+notification path.
+
+Notifications are per-operation invalidations. A sequential create, update and
+delete in one committed callback emits those three notices in order, even though
+the final row is absent. Several updates to one row are not coalesced. No notice
+is emitted while the callback remains pending. The helper awaits completion
+hooks; after a confirmed commit, a completion error cannot undo stored changes
+or notifications already emitted. A later operation's completion chain can flush
+the transaction's queue if an earlier chain failed before delivery. This does
+not guarantee delivery when every applicable chain is interrupted or the
+connection fails.
+
+Once queue draining begins, failure while delivering one queued notice does not
+prevent attempts for later notices. The call rejects with the first delivery
+failure and a committed outcome; it does not roll back stored data. The context
+of the operation draining the queue retains each failed attempt as
+`{ phase: 'socketioBroadcast', broadcastIndex, error }`, where `broadcastIndex`
+is zero-based in that transaction's captured queue. Keep that operation context
+when you need these diagnostics, just as for file cleanup diagnostics.
+
+The queue is consumed once. Failed notices are not automatically replayed by
+later completion chains. One notice can fail after delivery to some recipients;
+the plugin cannot establish who received it or undo those deliveries. Clients
+must be able to refetch authoritative data. Continuing with later notices is a
+best-effort attempt, not a delivery acknowledgement or durable retry mechanism.
+
+With overlapping writes, notification order follows their capture in the
+Socket.IO finish hook, which can differ from call-start order. Completion hook
+chains follow transaction enlistment order (reverse order on rollback). Await
+writes sequentially when their ordering matters to the application.
+
+Socket.IO preserves message ordering but does not provide durable delivery by
+default. Clients can miss notifications while disconnected; refetch on reconnect.
+See [Socket.IO delivery guarantees](https://socket.io/docs/v4/delivery-guarantees/).
+
+Optional Redis setup accepts the options used by Redis's `createClient`, for
+example `startSocketServer(server, { redis: { url: 'redis://localhost:6379' } })`.
+Install `redis` and `@socket.io/redis-adapter` when using it. Subscription state
+survives the JSON encoding used by the adapter's `fetchSockets` responses.
+The [real Redis tests](../contributing.md#real-databases-and-redis) exercise two Socket.IO servers
+sharing SQLite, in both storage modes, over polling and WebSocket. They cover
+committed CRUD and relationship notifications, row visibility, rollback and
+replacement of queued subscriptions. Further cases drop each publisher/subscriber
+connection on each server and verify delivery after reconnection. Startup tests
+cover bad credentials, missing sockets, exhausted retries, partial connection
+failure, preserved authentication and overlapping starts. Shutdown tests cover
+Socket.IO close, HTTP close and close during Redis reconnection.
+
+These tests do not establish durable delivery during an outage, Redis Cluster
+or Sentinel failover, or bounded cleanup when a network connection remains open
+but never answers. Redis shutdown drains queued commands; connection and retry
+timeouts remain part of the supplied Redis configuration.
+
+Socket.IO transport CORS is configured separately through
+`startSocketServer(server, { cors: ... })` or `transport.cors` in plugin options.
+It does not replace the HTTP [CORS plugin](24-cors.md) or authentication.

package/docs/GUIDE/29-writing-plugins.md

@@ -0,0 +1,278 @@
+---
+title: "Writing plugins"
+chapter: 29
+chapter_label: "29"
+---
+
+# 29. Writing plugins
+
+`JsonRestApi` installs plugins and runs resource methods through a small local
+runtime. This guide covers the public extension surface for plugin authors.
+
+Use this guide when you want to:
+
+- add resource-level behavior across many resources
+- add new resource methods
+- compile resource metadata into reusable runtime state
+- hook into query or write processing
+
+For the internal execution path, see [architecture](../architecture.md).
+
+## Minimal plugin shape
+
+These three blocks form a runnable example. Add them to the
+[starting script](03-running-example.md) after installing storage,
+before starting the server, on a fresh database. Later snippets are fragments
+for a plugin's `install` function.
+
+```javascript
+const MyPlugin = {
+  name: 'my-plugin',
+  dependencies: ['rest-api'],
+  install ({ addHook, addResourceMethod }) {
+    addHook('beforeSchemaValidate', 'trim-example-name', {}, ({ context }) => {
+      const attributes = context.inputRecord?.data?.attributes
+      if (typeof attributes?.name === 'string') attributes.name = attributes.name.trim()
+    })
+    addHook('resource:added', 'compile-my-plugin', {}, ({ context, scopes }) => {
+      const scope = scopes[context.scopeName]
+      scope.vars.myPlugin = { enabled: context.scopeOptions.exampleFlag === true }
+    })
+    addResourceMethod('describeExample', async ({ scopeName, scope }) => ({
+      scopeName, enabled: scope.vars.myPlugin.enabled
+    }))
+  }
+}
+await api.use(MyPlugin)
+```
+
+Install registration hooks before registering the resources they configure.
+
+```javascript
+await api.addResource('authors', {
+  schema: { name: { type: 'string', required: true } },
+  exampleFlag: true
+})
+await api.addResource('publishers', {
+  schema: { name: { type: 'string', required: true } }
+})
+await api.resources.authors.createKnexTable()
+await api.resources.publishers.createKnexTable()
+```
+
+The custom method is available on each resource, with resource-specific state.
+
+```javascript
+const authorDescription = await api.resources.authors.describeExample()
+const publisherDescription = await api.resources.publishers.describeExample()
+const createdAuthor = await api.resources.authors.post({
+  format: 'plain', data: { name: '  Ada  ' }
+})
+console.log(authorDescription, publisherDescription, createdAuthor)
+```
+
+The first result is `{ scopeName: 'authors', enabled: true }`; the second is
+`{ scopeName: 'publishers', enabled: false }`. The created author's `name` is
+`'Ada'`: the hook trims the declared attribute before attribute validation and
+storage.
+Export `MyPlugin` as a named
+export when moving the declaration into its own module.
+
+The usual pattern is:
+
+1. compile resource metadata during `resource:added`
+2. store normalized state in `scope.vars`
+3. use hooks or resource methods to apply behavior at runtime
+
+## Registration and hook ordering
+
+A plugin has a unique `name`, an `install` function and optional `dependencies`.
+Dependencies must already be installed. Each string names a required plugin;
+an array names alternatives, for example
+`dependencies: ['rest-api', ['rest-api-knex', 'rest-api-anyapi-knex']]`.
+Duplicate plugin installation rejects. Await setup calls sequentially and discard
+an instance if installation fails; installation is not transactional.
+
+The installation argument provides `addApiMethod(name, handler)`,
+`addResourceMethod(name, handler)` and `addHook(event, name, placement, handler)`.
+An API method name cannot replace an existing API member. Registering a shared
+resource method replaces the shared definition; resource-local overrides remain.
+Methods and hooks receive the common runtime envelope. Internal names `scope`,
+`scopeName` and `scopes` refer to resources; the public collection is `api.resources`.
+
+Use `{}` to append a hook, or specify one named `beforeFunction` or `afterFunction`
+anchor in the same event. A missing anchor rejects registration. Numeric order
+and per-plugin ordering options are unsupported. Handlers run sequentially over
+a snapshot of the list. Returning `false` stops that list; throw an error to reject
+an operation. The dispatcher propagates thrown values unchanged.
+
+Resource-local hooks use the same placement contract and run only for that
+resource. Register global `resource:added` hooks before adding resources; a
+resource's local hooks are installed after its setup event completes.
+
+## The stable extension points
+
+### `resource:added`
+
+Use this to inspect `scopeOptions`, validate configuration, and compile resource-specific runtime state into `scope.vars`.
+
+Typical uses:
+
+- initialize resource-specific extension state
+- validate non-schema resource options
+
+Declare fields during `schema:enrich` (or `computedSchema:enrich` for computed
+fields). Use `schema:compiled` for work that needs the compiled schema, such
+as autofilter configuration; do not add fields after compilation.
+
+Example:
+
+```js
+addHook('resource:added', 'compile-example', {}, ({ context, scopes }) => {
+  const scope = scopes[context.scopeName]
+  const options = context.scopeOptions || {}
+
+  scope.vars.example = {
+    flag: options.exampleFlag === true
+  }
+})
+```
+
+### `beforeSchemaValidate`
+
+Use this to normalize attribute values before attribute schema validation runs.
+Request document validation and relationship processing have already begun.
+Use the earlier `beforeProcessing` stage if an extension must change the input
+structure before those checks, and validate any untrusted shape it reads.
+
+Typical uses:
+
+- inject derived input values
+- apply attribute-level business checks
+
+This hook cannot make an otherwise rejected request document valid retroactively.
+
+The complete example above trims a declared string attribute here. It does not
+attempt to remove an unknown field after request validation has rejected it.
+For stage order and available context, use the
+[write lifecycle reference](13-hooks-and-lifecycle.md#resource-write-order).
+
+### `knexQueryFiltering`
+
+Use this to add query constraints to the generated Knex query.
+
+Typical uses:
+
+- automatic scoping
+- cross-table filters
+- custom public filter semantics
+
+The hook receives the current builder through `context.knexQuery.query`,
+along with its `scopeName` and `tableName`. It also runs for nested queries.
+Raw column names are unsafe to assume: ordinary resources can map logical fields
+to different columns, and canonical resources use storage slots and aliases.
+
+For equality scoping and write stamping, use
+[AutoFilterPlugin](16-autofiltering.md). For mandatory resource visibility,
+use [RowPolicyPlugin](17-row-policies.md), whose policy context supplies
+logical column/value translation and query-purpose metadata. For a public
+client-selected filter, use the
+[custom search filter contract](04-creating-and-querying.md).
+These mechanisms avoid duplicating storage translation in application hooks.
+
+A filtering hook does not turn an ad hoc SQL alias into a declared field;
+use the query-field contract below for selected or sortable derived values.
+
+### `addResourceMethod`
+
+Use this when a plugin needs a reusable method on every resource or selected resources.
+
+Example:
+
+```js
+addResourceMethod('introspect', async ({ vars }) => {
+  return {
+    tableName: vars.schemaInfo?.tableName,
+    fields: Object.keys(vars.schemaInfo?.schemaStructure || {})
+  }
+})
+```
+
+## The query-field seam
+
+`json-rest-api` now supports a small, explicit seam for **query-only read fields**.
+
+This is the seam used by `QueryProjectionsPlugin`, and it is the recommended pattern for plugins that need derived SQL-backed fields.
+
+### What a plugin should provide
+
+Declare query fields during the existing schema-enrichment stage:
+
+```js
+addHook('schema:enrich', 'declare-uppercase-name', {}, ({ context }) => {
+  if (context.scopeName !== 'authors') return
+  context.queryFields.uppercase_name = {
+    type: 'string',
+    sortable: true,
+    select: ({ knex, column }) => knex.raw('upper(??)', [column('first_name')])
+  }
+})
+```
+
+Install the hook before registering resources. Core normalizes the final map
+and rejects names that collide with attributes, computed fields, relationships
+or logical IDs. If another declaration hook replaces the map, order this hook
+after that one. The published definitions are available at
+`scope.vars.schemaInfo.queryFields`; do not mutate a separate `vars.queryFields`
+map. Canonical `addKnexFields` recompiles declarations and dependencies before
+publication. SQL callbacks are still evaluated with each query's context.
+
+### What core does with compiled query fields
+
+Core will:
+
+- include visible query fields by default in `get()` and `query()`
+- allow them in sparse fieldsets
+- allow them in sorting when the plugin marks them sortable
+- use them in cursor pagination with a stable `id` tie-breaker
+- carry them through included-resource selection
+- only expose declared query-field aliases on the AnyAPI path
+
+### What this seam is for
+
+Use it for:
+
+- SQL-backed projections
+- aggregate list fields
+- joined display labels
+- derived fields that must participate in `ORDER BY`
+
+Do **not** use it for:
+
+- writable fields
+- storage/schema ownership
+- migrations or introspection
+- normal response-only computed fields
+
+For the concrete projection example, see [Query Projections](15-query-projections.md).
+
+## Recommended plugin pattern
+
+When you add a new plugin feature, prefer this flow:
+
+1. read plugin options at install time
+2. declare schema metadata during its enrichment stage; use `resource:added` for other registration work
+3. store only normalized runtime state in `scope.vars`
+4. use hooks or resource methods to apply behavior
+
+That keeps the plugin declarative and avoids re-parsing configuration during requests.
+
+## Boundaries to keep clean
+
+- Keep **query-layer** extensions out of `schema`.
+- Choose input hooks according to the [write lifecycle](13-hooks-and-lifecycle.md#resource-write-order).
+- Keep **filtering** in `knexQueryFiltering`.
+- Keep **response-only** enrichment in `enrichAttributes`.
+- Do not rely on arbitrary internal helper filenames as public API.
+
+If a feature requires deeper integration than the public seams provide, treat that as a core extension discussion rather than a plugin hack.