json-rest-api 1.0.28 → 2.0.0

package/docs/GUIDE/30-backend-capabilities.md

@@ -0,0 +1,100 @@
+---
+title: "Backend capabilities and limits"
+chapter: 30
+chapter_label: "30"
+---
+
+# 30. Backend capabilities and limits
+
+This reference describes the v2 storage contracts. Follow the
+[migration guide](33-migrating-to-v2.md) before upgrading an application.
+Node 24 or later is required; verification runs on Node 24 only.
+
+## Database and storage choices
+
+| Combination | Verification and limits |
+| --- | --- |
+| SQLite through `better-sqlite3` | Verified with SQLite 3.49.2. Ordering helpers require SQLite 3.30 or later. Concurrent writers can fail with busy/snapshot conflicts; the library does not make SQLite a multi-writer database. |
+| PostgreSQL through `pg` | Verified on PostgreSQL 16. Native precision, constraints, transaction visibility and failure paths have dedicated checks. |
+| MySQL through `mysql2` | Verified on MySQL 8. DDL can implicitly commit and is not treated as an atomic rollbackable migration. |
+| MariaDB, MSSQL, Oracle and other Knex clients | Unverified, including dialects with branches in capability helpers. Those branches are not a support guarantee. |
+
+The library does not ship a generic remote-service or NoSQL CRUD backend. See
+[custom data sources](14-custom-resource-methods.md) for the distinction between
+custom service methods and a complete storage integration.
+
+The SQL matrix exercises ordinary resource tables and canonical AnyAPI storage.
+Canonical storage uses shared tables with tenant/resource predicates and opaque
+logical IDs; those IDs sort lexically. Ordinary numeric IDs sort numerically.
+Neither storage choice changes the database's transaction or collation behavior.
+Dedicated collation tests cover selected cases, not arbitrary database collations.
+Other server versions and Windows remain unverified.
+
+Per-parent include limits require window-function capability. Capability detection
+does not establish support for every query on an otherwise unverified dialect.
+See [contributing](../contributing.md#real-databases-and-redis) for native
+database verification commands. A passing SQLite suite is not native PostgreSQL/MySQL
+evidence, even when launched with a different storage-mode environment variable.
+
+## Temporal values
+
+Public date/dateTime inputs are strings; time inputs remain strings too. Built-in
+dateTime writes pass through JavaScript Date and support millisecond precision.
+They reject nonzero submillisecond digits. Declaring precision 6 does not enable
+microsecond writes through that conversion. Database string reads can retain
+microseconds; library PostgreSQL/MySQL queries request temporal strings to avoid
+loss through driver Date conversion. Caller-owned raw queries retain their own
+driver parsing behavior.
+
+Native PostgreSQL/MySQL time/dateTime columns accept declared precision 0–6.
+New columns default to precision 6; existing columns require a reviewed migration.
+MySQL native DATE/DATETIME writes reject years below 1000. Custom text mappings
+and synchronous serializers can support different storage representations, but
+must satisfy the chosen column, driver and read-normalization contracts.
+See [temporal migration details](33-migrating-to-v2.md#temporal-values-and-storage-serializers)
+for calendar ranges, UTC normalization, time formatting and old canonical slots.
+
+## Schema changes
+
+Generated diffs describe the declared schema versus the live table; they are not
+a migration-history engine. Review warnings and generated SQL. Unsupported
+changes can be omitted with warnings, and destructive `down` migrations are not
+generated automatically. SQLite check/enum changes can require a reviewed table
+rebuild. Native SET fields require an explicit MySQL target.
+
+AnyAPI field additions persist slot metadata; `alterKnexFields` is unsupported.
+Re-registering populated resources cannot remove or remap stored fields. Preserve
+slot maps and migrate stored data explicitly when changing layouts. Old canonical
+temporal storage needs the documented data migration, not just a package update.
+See [table helpers and migration limits](21-schema-and-migrations.md).
+
+## Transaction ownership and failures
+
+Use `api.transaction` to group library writes and raw SQL. Its callback receives
+a real Knex transaction, but the library owns completion and deferred hooks.
+Pass that handle to each participating operation and await each call. Unmanaged
+raw Knex transactions and child savepoints are rejected by library write methods.
+Raw transactions remain usable for direct SQL, low-level helpers and read-only
+API calls; each low-level schema/registry helper retains its documented contract.
+
+A rejected write can already be committed. Inspect the recorded transaction
+outcome; an unknown outcome requires reconciliation rather than blind replay or
+upload deletion. The library does not automatically retry conflicts. See
+[transactions and errors](33-migrating-to-v2.md#transactions-and-errors) for ownership,
+completion-hook behavior and migration examples.
+
+## Files, notifications and known gaps
+
+`LocalStorage` writes real files. Remote storage uses application-provided
+adapters implementing `upload` and `delete`. File cleanup is best effort; failures remain in
+diagnostic context for reconciliation. See [file storage and cleanup](26-file-uploads.md).
+
+Socket.IO has real WebSocket/polling and Redis integration checks. These are
+resource invalidations, not a durable event log or exactly-once delivery system;
+see the [Socket.IO contract](28-socketio.md).
+
+[Positioning](31-positioning.md) uses transaction-owned coordinator rows and has
+separate-connection insertion/move/rollback coverage on PostgreSQL and MySQL in
+both storage modes. Writes to one positioned resource are serialized. SQLite
+busy conflicts and stale PostgreSQL snapshots can still require an application
+retry; column collation and offline imports remain application responsibilities.

package/docs/GUIDE/31-positioning.md

@@ -0,0 +1,181 @@
+---
+title: "Positioning"
+chapter: 31
+chapter_label: "31"
+---
+
+# 31. Positioning
+
+The positioning plugin assigns fractional string keys for custom ordering within
+groups, such as sortable lists and drag-and-drop interfaces.
+
+Position allocation is coordinated by the database and participates in the
+resource operation's managed transaction. Separate-connection tests cover
+concurrent inserts, moves, group changes, rollback and earlier transaction
+snapshots on PostgreSQL and MySQL, in both ordinary and canonical storage.
+SQLite reports a busy/snapshot conflict when another connection holds its write
+lock; it does not silently allocate the same key.
+
+## Setup
+
+Install `fractional-indexing` alongside the ordinary Knex dependencies. The plugin
+is available through its package subpath, rather than the root exports:
+
+```javascript
+import { JsonRestApi, RestApiPlugin, RestApiKnexPlugin } from 'json-rest-api'
+import { PositioningPlugin } from 'json-rest-api/plugins/core/rest-api-positioning-plugin.js'
+
+// knex is application-owned; use bytewise position-column/database collation.
+const api = new JsonRestApi({ name: 'ordered-tasks' })
+await api.use(RestApiPlugin)
+await api.use(RestApiKnexPlugin, { knex })
+await api.use(PositioningPlugin, { filters: ['status'], autoIndex: false })
+
+await api.addResource('tasks', {
+  schema: {
+    title: { type: 'string', required: true },
+    status: { type: 'string', required: true, search: true },
+    position: { type: 'string', max: 255, maxLength: 255, search: true },
+    beforeId: { type: 'string', virtual: true }
+  }
+})
+await api.resources.tasks.createKnexTable()
+```
+
+Declare the position field yourself. The plugin validates its presence; it does
+not add the field or migrate an existing database. Install the plugin before
+registering participating resources. Each resource not excluded by configuration
+must also declare the grouping fields.
+
+Validation uses the compiled schema after `schema:enrich`, so a setup hook may
+supply the position field. Its final definition must be a string without a setter.
+
+Installation creates the internal `json_rest_api_positioning_locks` table when
+it is absent. It stores one coordinator row per physical resource, including
+the tenant/resource identity for canonical storage. Provision that table during
+deployment if the runtime database account cannot create tables:
+
+```javascript
+await knex.schema.createTable('json_rest_api_positioning_locks', table => {
+  table.string('resource', 64).primary()
+})
+```
+
+The plugin then needs ordinary read/write access to the table. Keep the table
+and its rows while applications are running; removing a live coordinator would
+allow writers to use different locks.
+
+## Resource calls
+
+```javascript
+const first = await api.resources.tasks.post({
+  data: { title: 'First task', status: 'todo' }
+})
+const last = await api.resources.tasks.post({
+  data: { title: 'Last task', status: 'todo', beforeId: null }
+})
+await api.resources.tasks.post({
+  data: { title: 'Middle task', status: 'todo', beforeId: last.id }
+})
+await api.resources.tasks.patch({
+  id: last.id,
+  data: { beforeId: first.id }
+})
+```
+
+`beforeId` identifies another item in the destination group. `null` means the end
+of that group, and `'FIRST'` means its beginning. New records without `beforeId`
+use `defaultPosition`. Updates within the same group retain their position when
+`beforeId` is omitted; changing a grouping field allocates a position in the
+destination group using `defaultPosition`. An explicit `beforeId` chooses a
+different placement. PUT creates and replacements follow the same rules.
+
+A self-targeted move within the same group keeps its position. Missing targets
+and targets outside the destination group fall back to appending. Explicit
+targets pass through the resource's normal GET visibility and permission checks;
+a permission error rejects the write. Manual position values are ignored.
+
+The plugin supplies position sorting when a query has no explicit sort. Position
+keys are strings, not numbers. Use a bytewise collation for the stored position
+column so ordinary queries display their intended order: PostgreSQL `C`, MySQL
+`utf8mb4_bin`, or SQLite `BINARY`, for example. The allocator uses explicit binary
+comparisons, but ordinary sorting and cursor predicates still use the column's
+database collation. A case-insensitive collation can display or paginate the
+wrong order. Do not substitute arbitrary integer strings for valid fractional keys.
+Grouping separates ordering sequences; it does not provide authorization.
+Use [row policies](17-row-policies.md) for mandatory visibility restrictions.
+
+## HTTP input
+
+The connectors accept JSON:API documents. A request to move an existing task
+uses `PATCH /api/tasks/42` with `Content-Type: application/vnd.api+json`:
+
+```json
+{
+  "data": {
+    "type": "tasks",
+    "id": "42",
+    "attributes": { "beforeId": "17" }
+  }
+}
+```
+
+Plain records belong in programmatic `data` arguments; they are not the
+HTTP document contract.
+
+## Configuration and limits
+
+| Option | Default | Behavior |
+| --- | --- | --- |
+| `field` | `position` | Declared string field holding the ordering key |
+| `filters` | `[]` | Fields defining independent position groups |
+| `excludeResources` | `['system_migrations', 'system_logs']` | Resources exempt from positioning |
+| `beforeIdField` | `beforeId` | Request attribute containing the placement target |
+| `defaultPosition` | `last` | Default placement for new records; `first` is also supported |
+| `strategy` | `fractional` | The implemented strategy; `integer` rejects |
+| `autoIndex` | `true` | Attempts an ordinary-table index during resource registration |
+
+Automatic index creation skips canonical storage and tables that do not yet
+exist. Failures are logged rather than aborting setup. Manage required indexes
+through reviewed [schema migrations](21-schema-and-migrations.md); do not assume
+that enabling `autoIndex` created an index.
+
+`api.positioning.getConfig()` returns the configuration, and
+`api.positioning.isEnabled(resourceName)` reports whether a registered resource
+participates. The old no-op `api.positioning.reorder()` helper has been removed;
+use resource PATCH operations or [atomic bulk PATCH](27-bulk-operations.md).
+The formerly inert `rebalanceThreshold` option now rejects.
+
+There is no automatic rebalancing. Keys that exceed the tighter of the schema's
+`max` and `maxLength` (default physical length: 255) reject before storage, so a
+database cannot silently truncate a newly allocated key. The
+`rebalancePositions()` utility calculates replacement keys in memory; it does
+not persist them. `getUnpositionedItems()` and `assignInitialPositions()` support
+offline data migrations, using actual fractional-key validation.
+
+## Concurrency and transaction boundaries
+
+Positioned POST/PUT/PATCH operations lock one coordinator row before processing
+the request. All position groups within that resource share the lock. This
+deliberately serializes positioned writes to keep empty groups and moves between
+groups safe without multiple group locks. The lock lasts until the owning
+transaction commits or rolls back, including when the operation borrows an
+`api.transaction()` handle. Keep those transactions short.
+
+PostgreSQL and MySQL wait for the earlier writer and then allocate against
+current rows. PostgreSQL repeatable-read transactions with an older snapshot
+can reject with a serialization conflict; SQLite can reject with a busy/snapshot
+conflict. Deadlocks, timeouts and connection failures remain ordinary transaction
+failures. No hooks or external effects are replayed automatically. When an
+application retries a failed move, it must start a fresh operation/transaction.
+
+The coordination guarantee covers writers using this plugin against the same
+database and lock table. Direct SQL imports, manual rebalancing and other writers
+must not change positions concurrently. Do not add a setter to the managed
+position field or change position/group fields after allocation. The plugin
+retains normal resource permissions, validation and commit/rollback handling.
+
+For existing data, plan a separate migration with valid keys, explicit ordering
+and database-appropriate collation. Back up data and inspect the result before
+enabling the plugin; registration does not convert existing positions or repair
+duplicate keys. Existing groups must start with valid, distinct keys.

package/docs/GUIDE/{GUIDE_Y_Appendices.md → 32-date-and-time.md}

@@ -1,4 +1,10 @@
-# Appendices: Date and Time Handling
+---
+title: "Date and time"
+chapter: 32
+chapter_label: "32"
+---
+
+# 32. Date and time
 
 ## Overview
 
@@ -39,15 +45,25 @@ The API supports three string temporal types and two numeric epoch types in sche
 
 ## Schema Definition
 
-Define date/time fields in your schema like this:
+The following three executable blocks form one scenario. Add them to the
+[starting script](03-running-example.md) after installing storage and
+before starting the server, on a fresh database. Other snippets are illustrative
+payloads or configuration fragments.
 
 ```javascript
-const articleSchema = {
-  publishedDate: { type: 'date', required: true },
-  createdAt: { type: 'dateTime', temporalPrecision: 3, defaultTo: () => new Date().toISOString() },
-  updatedAt: { type: 'dateTime', temporalPrecision: 3, defaultTo: () => new Date().toISOString() },
-  dailyPostTime: { type: 'time', nullable: true }
-};
+await api.addResource('articles', {
+  schema: {
+    title: { type: 'string', required: true },
+    publishedDate: { type: 'date', required: true, search: true },
+    createdAt: { type: 'dateTime', temporalPrecision: 3, required: true },
+    dailyPostTime: { type: 'time', nullable: true },
+    publishedEpoch: { type: 'epochMilliseconds', required: true }
+  },
+  searchSchema: {
+    createdBetween: { type: 'array', actualField: 'createdAt', filterOperator: 'between' }
+  }
+})
+await api.resources.articles.createKnexTable()
 ```
 
 `Date.prototype.toISOString()` always emits three fractional-second digits, so
@@ -63,19 +79,28 @@ than silently truncating it.
 The API validates and normalizes date/time inputs:
 
 ```javascript
-// POST /api/articles
-{
-  "data": {
-    "type": "articles",
-    "attributes": {
-      "publishedDate": "2024-01-15",           // Valid date
-      "createdAt": "2024-01-15T14:30:00Z",     // Valid dateTime (ISO 8601)
-      "dailyPostTime": "14:30:00"              // Valid time
-    }
+const januaryArticle = await api.resources.articles.post({
+  data: {
+    title: 'January', publishedDate: '2024-01-15',
+    createdAt: '2024-01-15T22:30:00+08:00', dailyPostTime: '14:30:00',
+    publishedEpoch: '1705329000000'
   }
-}
+})
+await api.resources.articles.post({
+  data: {
+    title: 'February', publishedDate: '2024-02-01',
+    createdAt: '2024-02-01T00:00:00Z', dailyPostTime: null,
+    publishedEpoch: 1706745600000
+  }
+})
+console.log(januaryArticle)
 ```
 
+The first result retains date `2024-01-15` and time `14:30:00`, normalizes
+createdAt to `2024-01-15T14:30:00.000Z`, and returns publishedEpoch as the number
+`1705329000000`. Its dateTime offset describes the same instant as UTC; callers
+are not required to send only `Z` inputs.
+
 **Accepted Input Formats:**
 - **date**: a real calendar date in `YYYY-MM-DD` form
 - **dateTime**: an RFC 3339 string with seconds and an explicit timezone, such as `2024-01-15T14:30:00Z` or `2024-01-15T22:30:00+08:00`
@@ -86,13 +111,27 @@ JavaScript `Date` objects, SQL datetime strings, Unix timestamps, locale text, a
 **Storage Format:**
 Schema validation preserves valid JSON temporal strings exactly. The built-in storage adapters convert `date` and `dateTime` strings to database-native `Date` values at the database boundary, while `time` remains a string. A custom field-level `storage.serialize` function receives the validated string and owns any custom database conversion.
 
+The built-in `dateTime` conversion supports millisecond precision. A value such
+as `2026-09-01T10:20:30.123456Z` fails with HTTP 422 before the write, because
+converting it to `Date` would lose information. Trailing zeroes beyond three
+digits are accepted (`.123000Z` represents exactly `.123Z`). To retain finer
+precision in table-backed storage, use a custom `storage.serialize` function
+and a database column and driver that preserve it. The same serializer is used
+for filters and cursor comparisons, with `operation: 'filter'` and `context: null`.
+
+Response normalization preserves fractional digits returned as strings by
+storage, up to the field's declared precision, including during conversion of
+timezone offsets to UTC. It also converts valid epoch strings and `bigint`
+values from database drivers to JSON numbers, rejecting invalid or out-of-range
+epoch data.
+
 ## Output Normalization
 
 ### Database to API Response
 
 All date/time values are normalized when returned from the API:
 
-```javascript
+```js
 // GET /api/articles/123
 {
   "data": {
@@ -115,8 +154,9 @@ All date/time values are normalized when returned from the API:
 3. **UTC Assumption**: MySQL DATETIME values (which lack timezone info) are assumed to be UTC
 
 Normalization runs once after the database read and again at the final response
-boundary. The final pass covers getters, computed fields, and finishing hooks
-that produce native `Date` values. A malformed non-null temporal value is not
+boundary. The final pass covers getters, computed fields, query projections,
+and finishing hooks that produce native `Date` values, including nested
+relationships in plain write responses. A malformed non-null temporal value is not
 reported as `null`; it fails with `REST_API_TEMPORAL_DATA_INVALID` and HTTP 500
 so stored-data or enrichment defects remain visible.
 
@@ -135,7 +175,7 @@ so stored-data or enrichment defects remain visible.
 ## Best Practices
 
 ### 1. Always Store in UTC
-```javascript
+```js
 // Good: Store timestamps in UTC
 const article = {
   createdAt: new Date().toISOString() // "2024-01-15T14:30:00.000Z"
@@ -153,21 +193,40 @@ const article = {
 - Use `time` for recurring daily events
 
 ### 3. Timezone Handling
-- Send all dateTime values to the API in UTC
+- Send dateTime values with an explicit timezone; UTC is convenient, and valid offsets are accepted
 - The API always returns dateTime values in UTC (with 'Z' suffix)
 - Handle timezone conversion in your client application
 
 ### 4. Filtering and Querying
-When filtering by dates, use ISO 8601 format:
+
+Declare public filter names in the schema or search schema, then use
+`queryParams.filters` in programmatic calls. Temporal filter values use the
+same declared field conversion as writes.
 
 ```javascript
-// Filter articles published after a date
-GET /api/articles?filters[publishedDate][$gte]=2024-01-01
+const onDate = await api.resources.articles.query({
+  queryParams: { filters: { publishedDate: '2024-01-15' } }
+})
+const inJanuary = await api.resources.articles.query({
+  queryParams: {
+    filters: { createdBetween: ['2024-01-01T00:00:00Z', '2024-01-31T23:59:59.999Z'] }
+  }
+})
+console.log(onDate.data.map(article => article.title), inJanuary.data.map(article => article.title))
+```
+
+Both queries return January only. `between` includes both boundaries. The
+corresponding HTTP equality query uses the singular `filter` key:
 
-// Filter by datetime range
-GET /api/articles?filters[createdAt][$gte]=2024-01-01T00:00:00Z&filters[createdAt][$lt]=2024-02-01T00:00:00Z
+```http
+GET /api/articles?filter[publishedDate]=2024-01-15
 ```
 
+Use declared aliases such as `createdBetween`; arbitrary `$gte`/`$lt` objects
+are not the public filter grammar. See
+[searching](04-creating-and-querying.md) for custom ranges and
+[backend limits](30-backend-capabilities.md#temporal-values) for precision rules.
+
 ## Migration Considerations
 
 ### From Existing Systems
@@ -183,18 +242,18 @@ If migrating from a system that stores dates differently:
 For optimal date handling, configure your database connection:
 
 **MySQL** (in Knex config):
-```javascript
+```js
 {
   client: 'mysql2',
   connection: {
     // ... other config
-    timezone: 'UTC'
+    timezone: 'Z'
   }
 }
 ```
 
 **PostgreSQL** (in Knex config):
-```javascript
+```js
 {
   client: 'pg',
   connection: {