json-rest-api 3.0.0-rc.0 → 3.0.1

package/docs/GUIDE/33-migrating-to-v2.md

@@ -2031,8 +2031,8 @@ empty cursor rejects before SQL. Follow generated links to retain filters,
 fieldsets and the effective size.
 
 `enablePaginationCounts` controls totals for offset pages. With counts disabled,
-offset metadata contains `page` and `pageSize`; it does not promise totals or
-`hasMore`. Cursor pages use one extra row for `hasMore` and never issue a count
+offset metadata contains `page` and `pageSize` (and, since 3.0, `hasMore`, from
+one extra row); it does not promise totals. Cursor pages use one extra row for `hasMore` and never issue a count
 query, regardless of this setting. Ordinary storage previously issued a count
 on the first cursor page and discarded its result; that extra query and its
 count-filter hook invocation have been removed. Collection predicates still

package/docs/GUIDE/34-scoped-reads-and-locks.md

@@ -15,7 +15,7 @@ it does not need hand-written Knex that skips them:
 - `scopedQuery`, a read-only Knex builder already limited to the caller's rows;
 - `queryAll`, every matching row in cursor pages;
 - window includes, the newest few related rows per parent;
-- `lockByIds` and `claim`, row locks inside a transaction;
+- `lockByIds`, `lockWhere` and `claim`, row locks inside a transaction;
 - `lockVersionField`, a counter every update advances;
 - system contexts, for jobs that work across every owner.
 
@@ -149,9 +149,33 @@ shared: every owner reads them. The field must be `nullable`.
 | Operation | Sees |
 | --- | --- |
 | `query`, `get`, includes, `aggregate`, `scopedQuery` (`purpose: 'report'`) | the owner's rows and shared rows |
-| many-row writes, `lockByIds`, `claim`, `scopedQuery` with `purpose` `'bulk-write'`, `'lock'` or `'claim'` | only the owner's rows |
-| single `patch`, `put`, `delete` of a shared row | refused as `forbidden` outside a system context |
-| `post` | stamps the owner; a create whose owner resolves to NULL is `forbidden` |
+| many-row writes, `lockByIds`, `lockWhere`, `claim`, `scopedQuery` with `purpose` `'bulk-write'`, `'lock'` or `'claim'` | only the owner's rows |
+| single `patch`, `put`, `delete` of a shared row, and `postRelationship` or `deleteRelationship` on it | refused as `forbidden` outside a system context |
+| `post` | stamps the owner; a create that would leave the field NULL is `forbidden` |
+
+A filter with `required: false` may not resolve, for example for a caller
+without a workspace. Reads are then not narrowed by it, but writes, locks and
+claims still skip the shared rows, and a create outside a system context must
+name an owner.
+
+`deleteRelationship` on a many-to-many relationship leaves links to targets
+the caller cannot see, as it leaves links to missing targets: when the target
+type has ownership filters or a row policy, it reads which targets are visible
+first (one read per type).
+
+A read shows a belongs-to link whose target the caller cannot see as
+unassigned (`data: null`), so a caller that writes the record back, as a form
+sends what it read, sends null for a link it never saw. `patch`, `put` and
+the many-row patches therefore keep such a link: a belongs-to (or polymorphic
+belongs-to) the write sets to null, whose stored target exists but is hidden
+from the writer, stays as stored, and the rest of the write applies. Linking
+a visible target, and clearing a link to a visible or missing target, apply as
+asked. A write that names a target is checked as before, even when it names
+the stored one: accepting a hidden target because it is unchanged would tell
+the writer which hidden record the link points to, since any other hidden id
+is refused. Only a write that clears a link pays for the check: one read of the
+targets' visibility, and one more of the hidden targets' existence. A
+many-row patch with such a row reads its rows first.
 
 Create and change shared rows with a [system context](#system-contexts).
 `api.autofilter.getScopeConfig(...)` reports `shared` for each filter, and
@@ -159,12 +183,17 @@ Create and change shared rows with a [system context](#system-contexts).
 
 ## scopedQuery
 
-`scopedQuery(params, context)` resolves to `{ query }`: a Knex builder on the
-resource's table with the caller's ownership filters, row policy and the given
+`scopedQuery(params, context)` resolves to `{ query }`: a Knex builder over the
+resource's rows with the caller's ownership filters, row policy and the given
 search `filters` already applied. (It is wrapped in an object because a Knex
 builder is thenable and would run if an async method returned it.) Use it for
 reports that `aggregate` cannot express, and as a subquery.
 
+The builder selects from a derived table, `(SELECT alias.* FROM table AS alias
+WHERE …) AS alias`, so conditions you add, `orWhere` and `whereRaw('a OR b')`
+included, can only narrow the scoped rows. Refer to columns as `alias.column`
+or by bare name.
+
 | Parameter | Default | Meaning |
 | --- | --- | --- |
 | `filters` | | Search filters, validated as `query` validates them |
@@ -183,11 +212,20 @@ const totals = await query
 
 The caller needs `query` permission on the resource. The builder is
 SELECT-only: `insert`, `update`, `delete`, `forUpdate` and the other locking
-reads, table changes and the methods that clear existing predicates throw
-`RestApiScopedQueryReadOnlyError`, on clones too. A table you join is not
-scoped by the builder; join only what the caller may see, or scope it with its
-own `scopedQuery`. `readOnlyQueryBuilder(builder, { resourceType })` is
-exported for builders your own code produces.
+reads, table changes (`from`, `fromRaw`, `table`), the methods that clear
+existing predicates, set operations (`union`, `unionAll`, `intersect`,
+`except`), common table expressions (`with`, `withRecursive`,
+`withMaterialized`), access to its Knex `client`, replacing or deleting its
+statement lists (`_statements`, `_single`) or its statement kind, and defining
+properties throw `RestApiScopedQueryReadOnlyError`, on clones too. Read from
+outside Knex's own methods, `_statements` is a frozen copy, so it cannot be
+changed in place either, and `_single` is a copy, so changing it changes no
+query. The builder
+guards against mistakes, not hostile code: code that means to bypass it can use
+Knex directly. A table you join is not scoped by the builder; join only what
+the caller may see, or scope it with its own `scopedQuery`.
+`readOnlyQueryBuilder(builder, { resourceType })` is exported for builders your
+own code produces.
 
 ## queryAll
 
@@ -244,16 +282,90 @@ and MariaDB 10.2+ do. AnyAPI storage applies the same rule.
 
 ## Row locks
 
-`lockByIds` and `claim` lock rows until the transaction ends. Both refuse to run
-outside a transaction (`RestApiLockOutsideTransactionError`); pass one from
-`api.transaction`. They return the records as the locking read saw them.
+`lockByIds`, `lockWhere` and `claim` lock rows until the transaction ends. All
+three refuse to run outside a transaction (`RestApiLockOutsideTransactionError`);
+pass one from `api.transaction`. They return the records as the locking read saw
+them, through the read pipeline of `query`: `checkDataPermissions` runs for each
+row as for `get`, then `enrichRecord`, computed fields and the `finish` and
+`finishQuery` hooks, which see `method: 'query'` and `lock: true`.
+
+The locking read is the lock's first statement on the resource's rows and
+carries the caller's ownership and row-policy predicates itself; the library
+makes no plain read of those rows before it. On MySQL and MariaDB this matters:
+under REPEATABLE READ the first plain read of a transaction fixes its
+snapshot, so a plain read before a lock wait would hide from every later read
+in the transaction what the lock holder committed while it waited (and with
+`innodb_snapshot_isolation=ON` the lock itself would fail with
+`ER_CHECKREAD`). After a lock, plain reads in the same transaction see what the
+previous holder committed, as long as the transaction made no plain read
+before the lock. Some plain reads can still come first or run inside the
+statement, and start the snapshot: subqueries of a row policy (InnoDB reads a
+locking read's subqueries without locks, from the snapshot), the scope
+subquery of a selection write's locking read on MySQL (`patchWhere`,
+`deleteWhere`), a `checkPermissions` or `knexQueryFiltering` hook that queries
+the database, and the first `lockWhere` for a MySQL table and set of filtered
+columns, which reads the table's indexes (below). Where the snapshot matters, keep those reads out of the lock's
+path, or make them before the transaction. For the same reason the query must
+stay a plain `SELECT` on the resource's table: a row policy or
+`knexQueryFiltering` hook that adds joins, a common table expression or a set
+operation, or replaces the source table, is refused for all three (rule
+`lock_single_table` or `claim_single_table`), because the database would lock
+the joined rows too or lock nothing at all.
 
 `lockByIds({ ids, transaction, missing, format, queryParams: { fields } })`
 locks the listed rows the caller can see, in id order, so two transactions that
-lock overlapping sets cannot deadlock on each other. Hidden or missing ids throw
+lock overlapping sets cannot deadlock on each other. It locks 1,000 ids per
+statement; longer lists are sorted first (integer ids numerically, other ids
+by code unit, the database's order for integer keys and binary collations), so
+they too lock in one order. Hidden or missing ids throw
 `RestApiRecordsNotFoundError`, or are left out with `missing: 'skip'`. The
-locked rows are read again under the lock, so they show the latest committed
-values even when the transaction began earlier.
+locked rows show the latest committed values even when the transaction began
+earlier.
+
+On MySQL and MariaDB, InnoDB locks each listed primary-key row before it
+evaluates the ownership and row-policy predicates, and under REPEATABLE READ
+holds that lock until commit. A listed id the caller cannot see is therefore
+still locked, and waited for when another transaction holds it, although it
+is not returned: its lock is a short denial of service against the row's owner
+and its wait a timing signal that the row exists. This is how
+`WHERE id = ? AND owner = ? FOR UPDATE` behaves on InnoDB. Pass `lockByIds` ids
+from a scoped read or from a route that has already checked the caller may use
+them, never a raw list from a request.
+
+`lockWhere({ filters, transaction, format, queryParams: { fields } })` locks
+every row the filters select that the caller can see, in id order, waiting for
+transactions that hold them, and returns them. A row that starts to match while
+it waits is locked and returned too: InnoDB's locking reads see the latest rows,
+and on PostgreSQL, whose statements only find rows committed when they began,
+`lockWhere` reads again under the locks it holds until no new row appears. It
+locks at most 1,000 rows and throws `RestApiRowLimitError` when more match. Use
+it, not `scopedQuery`, to lock by filter: the scoped builder is a read-only
+derived table that cannot lock.
+
+On MySQL and MariaDB, InnoDB locks every row a locking read examines, and every
+gap between them, until commit. Ordering by id could make it walk the primary
+key across every owner's rows, so there `lockWhere` has no `ORDER BY` and
+forces (`FORCE INDEX`) an index that keeps the read inside the caller's rows;
+rows come in that index's order. The index's leading columns must be the whole
+ownership (autofilter) columns the read is limited to (a prefix index on them
+does not count), or, when no ownership condition applies, it must start with a
+column the filters compare. Among those, an index whose next column the filters
+compare is preferred. Invisible (MySQL) and ignored (MariaDB) indexes cannot be
+forced and do not count. Without such an index `lockWhere` throws
+`RestApiBulkWriteUnsupportedError` before locking anything. Each call reads the
+connection's schema with `SELECT DATABASE()`, which reads no table, so one pool
+can serve several databases. The first call for a schema, table and set of
+columns also reads the table's indexes, in the transaction, and the index it
+picks is remembered while the process runs: restart after changing the
+indexes of a table that `lockWhere` uses.
+
+No ownership condition applies to a resource without ownership fields, to a
+system context, or to a resource that has only a row policy, which is not an
+ownership condition. There the forced index starts with a filtered column, so
+the read examines and locks every row the filters match, the rows of other
+policy owners included, although it returns only the rows the policy shows.
+Give such a resource ownership fields, or lock its rows with `lockByIds` on ids
+from a scoped read.
 
 `claim({ filters, sort, limit, transaction, format, queryParams: { fields } })`
 takes up to `limit` matching rows (default 1, at most 1,000) that no other
@@ -263,14 +375,31 @@ queue. Change the claimed rows in the same transaction, as the example does.
 A row policy that joins other tables is refused for `claim`, because the
 database would lock the joined rows too.
 
-| Database | `lockByIds` | `claim` |
-| --- | --- | --- |
-| PostgreSQL | `FOR UPDATE` on the visible ids | `FOR UPDATE SKIP LOCKED` |
-| MySQL 8, MariaDB 10.6+ | `FOR UPDATE` on the visible ids | reads candidates, then locks windows of them by primary key with `SKIP LOCKED`, because InnoDB locks every row a locking read examines |
-| SQLite | a scoped read | a scoped read |
-
-SQLite allows one writer at a time, so a write transaction already excludes
-every other one; its locks are ordinary scoped reads.
+| Database | `lockByIds` | `lockWhere` | `claim` |
+| --- | --- | --- | --- |
+| PostgreSQL | `FOR UPDATE` on the visible ids | `FOR UPDATE`, read again until no new row appears | `FOR UPDATE SKIP LOCKED` |
+| MySQL 8, MariaDB 10.6+ | `FOR UPDATE` on the listed ids, hidden ones included (above) | `FOR UPDATE` without `ORDER BY`, forced onto an index that starts with the ownership or a filtered column | reads candidates, then locks windows of them by primary key on the bare table with `SKIP LOCKED` (InnoDB locks every row a locking read examines), and reads the held rows again under the lock to drop any that stopped matching |
+| SQLite | the write lock, then a scoped read | the write lock, then a scoped read | the write lock, then a scoped read |
+
+On MySQL and MariaDB the candidate read of `claim` is a plain read, the only way
+to choose rows in sort order without locking every row the sort examines, so it
+starts the transaction's snapshot. `claim` never waits for a lock, so that
+snapshot misses only what commits during the claim itself; with
+`innodb_snapshot_isolation=ON` a candidate that another transaction changes
+between the two reads makes the claim fail with `ER_CHECKREAD`, and the caller
+retries.
+
+SQLite allows one writer at a time, but a transaction that has only read holds a
+shared lock, which does not keep another connection from writing, and its own
+later write could then fail with `SQLITE_BUSY` halfway through the work. So on
+SQLite each lock first runs a write that changes nothing
+(`UPDATE … SET id = id WHERE 0 = 1`), which takes the database's write lock for
+the rest of the transaction, once per transaction, and then reads the rows. The
+caller's transaction can stay a deferred `BEGIN`. Because the lock is a write,
+it fails with `SQLITE_READONLY` on a read-only connection. In WAL mode, a
+transaction that has already read keeps that snapshot, and if another
+connection has written since, taking the write lock fails with
+`SQLITE_BUSY_SNAPSHOT`; roll back and run the transaction again.
 
 ## Lock versions
 
@@ -317,7 +446,17 @@ With a system context:
 - autofilter applies no ownership filters, and creates must set every ownership
   field explicitly (NULL creates a shared row);
 - row policies receive `system: true` and decide what it means for them;
-- permission checks run as usual; `isSystemContext(context)` lets them tell.
+- permission checks run as usual; `isSystemContext(context)` lets them tell;
+- `aggregate` may group by, and count distinct, a hidden ownership field, so a
+  job can find its units of work, then run each under a context for that owner:
+
+  ```js
+  const { data } = await api.resources.entries.aggregate({
+    groupBy: ['workspace_id'],
+    measures: { rows: { count: '*' } },
+    queryParams: { filters: { status: 'open' } }
+  }, createSystemContext('find workspaces with open entries'))
+  ```
 
 A context with `system: true` but no marker throws
 `REST_API_SYSTEM_CONTEXT_FORGED`. Create system contexts only in server code

package/docs/GUIDE/35-migrating-to-v3.md

@@ -20,7 +20,8 @@ bulk options are rejected with a message that names their replacement.
   and compare-and-set (`onlyIf`). See [many-row writes](27-many-row-writes.md).
 - **Scoped reads:** `scopedQuery`, a read-only Knex builder limited to the
   caller's rows, and `queryAll`, every matching row in cursor pages.
-- **Row locks:** `lockByIds` and `claim` (`SKIP LOCKED`), inside a transaction.
+- **Row locks:** `lockByIds`, `lockWhere` and `claim` (`SKIP LOCKED`), inside a
+  transaction.
 - **Lock versions:** `lockVersionField`, a counter every update advances.
 - **Shared rows:** an autofilter filter with `shared: true` lets every owner
   read rows whose owner field is NULL.
@@ -31,11 +32,64 @@ bulk options are rejected with a message that names their replacement.
   registrations take `{ batch: true }`.
 - **Aggregates:** measures take a `where` condition; `groupBy` takes date
   buckets (`day`, `week`, `month`, `year`) with a time zone; date fields with a
-  storage serializer can be bucketed and take `min` and `max`. See
+  storage serializer can be bucketed and take `min` and `max`. In a system
+  context, a hidden ownership field may be grouped and counted distinct. See
   [AGGREGATE](../API.md#aggregate---grouped-totals).
-
-The last five are described in
-[scoped reads, locks and system work](34-scoped-reads-and-locks.md).
+- **One read for `get`:** a resource with no `checkPermissions`,
+  `checkDataPermissions`, `checkDataPermissionsGet`, `beforeData` or
+  `beforeDataGet` hook reads its record in one statement; such a get no
+  longer sets `context.minimalRecord` (read `context.record` in later hooks).
+- **Skipping writes in one statement:** `patchMany`, `patchWhereIds` and
+  `deleteMany` with `missing: 'skip'` put the scope in the UPDATE or DELETE
+  instead of reading the visible ids first, when no permission or
+  before-data-call batch hook needs those rows (not on MySQL and MariaDB,
+  where InnoDB would lock every listed row before checking the scope; there a
+  patch of one visible row runs its UPDATE without a locking read first).
+- **Fewer reads before single writes:** on table and AnyAPI storage, `patch`
+  and `delete` read their row once, scoped, before the UPDATE or DELETE, and a
+  `put` that replaces a row reads it twice (whether it exists, then scoped)
+  instead of three times. The unscoped read that checked the row still existed
+  is gone; the write's own row count answers a row that vanished in between
+  (`not_found`, as before).
+- **Single writes keep to their owner:** on table and AnyAPI storage, the
+  UPDATE or DELETE of a single `patch`, `put` or `delete` carries the
+  ownership (autofilter) conditions its scoped read found, as many-row writes
+  already did. A row that another writer moved to a different owner after the
+  read answers `not_found` instead of being written. On AnyAPI storage a row
+  that vanished after the read now answers `not_found` too, where the write
+  used to change nothing and report success.
+- **Links to hidden targets survive a write-back:** a `patch`, `put`,
+  many-row patch or merge upsert that sets a belongs-to to null keeps the
+  stored link when its target exists but is hidden from the writer, as the read
+  showed the link as unassigned. It used to clear it. Clear such links in a
+  system context.
+- **Polymorphic links are set through the relationship only:** the type and id
+  fields behind a polymorphic belongs-to (`owner_type`, `owner_id`) are refused
+  in attributes, as foreign keys already were (rule
+  `foreign_key_in_attributes`), so a write cannot skip the target check or half
+  clear a link. Send `owner: { _type, id }` (plain) or `relationships.owner`
+  (JSON:API) instead.
+- **Selection writes:** `patchWhere` and `deleteWhere` carry the scope and the
+  filters in every UPDATE or DELETE and return the rows they wrote; a row that
+  vanishes or stops matching in the meantime is left out, never an error.
+- **Counts:** `returning: 'count'` on every many-row write returns
+  `{ meta: { count } }`, the number of rows written, without reading rows back.
+- **Upserts on the id:** on a resource with no ownership fields and no row
+  policy, `onConflict: { keys: ['id'], action }` (or the `idProperty`, such as
+  an email natural key) creates the rows that do not exist yet, atomically.
+- **Integer fields over BIGINT columns read as numbers:** an `integer` or
+  `number` field whose driver returns a string (PostgreSQL's BIGINT, MySQL's
+  BIGINT past 2^53) reads as a number when a JavaScript number holds it exactly,
+  on every read and write response, as aggregates already did. It used to stay
+  a string, such as `"4400"`. A longer integer, or a fraction a number cannot
+  hold exactly, keeps its digits as a string, where an aggregate used to round
+  it; decimal fields (`precision` and `scale`) keep their strings.
+- **Numbered pages without counts:** with `enablePaginationCounts: false` an
+  offset page reads one extra row and reports `hasMore` and a `next` link,
+  still without a count query.
+
+Scoped reads, row locks, lock versions, shared rows and system contexts are
+described in [scoped reads, locks and system work](34-scoped-reads-and-locks.md).
 
 ## Many-row writes replace the bulk methods
 
@@ -50,10 +104,11 @@ the per-row checks in memory and one INSERT, UPDATE or DELETE per chunk.
 | `bulkPatch({ operations: [{ id, document: { data } }] })` | `patchMany({ document: { data: [{ type, id, attributes }] } })` |
 | the same change for many ids | `patchWhereIds({ ids, data })` or `patchWhere({ filters, data })` |
 | `bulkDelete({ ids })` | `deleteMany({ ids })` or `deleteWhere({ filters })` |
+| the same change for, or deletion of, every row | `patchWhere({ all: true, data })`, `deleteWhere({ all: true })`: without `filters`, `all: true` is required |
 | `atomic: true` | remove: every call is all-or-nothing |
 | `atomic: false` | remove; for rows that must succeed independently, call single writes in a loop |
 | `expectedVersion`, `expectedVersions` | removed: resources with a `versionField` refuse many-row writes. Use single writes with `expectedVersion`, or a `lockVersionField` with `onlyIf` |
-| result `{ meta: { total, succeeded, failed, atomic }, errors, data }` | nothing by default; `returning: 'minimal'` gives `{ data: [{ type, id }] }`, `'full'` gives `{ data: [records] }` |
+| result `{ meta: { total, succeeded, failed, atomic }, errors, data }` | nothing by default; `returning: 'minimal'` gives `{ data: [{ type, id }] }`, `'full'` gives `{ data: [records] }`, `'count'` gives `{ meta: { count } }` |
 | `errors[i].index` for a failed entry | the call throws; a validation error names the row in `details.index`, and its field paths start with `data.<index>.` |
 | `meta.deleted` | `deleteMany({ ids, returning: 'minimal' }).data` |
 | default `returning` of the resource | `'none'` unless the call says otherwise |
@@ -148,6 +203,67 @@ new methods; their parameter types are exported from the package root
 `RecordsResult` and others). `BulkOperationsPluginOptions` has only
 `maxBulkOperations`.
 
+## Objects in single-value fields
+
+Only `object` and `array` fields store a structure. A write, filter or cursor
+that gives any other field an object or array, as a `none` or `blob` field
+accepted from JSON, now fails with `RestApiValidationError` (rule
+`single_value`) before any SQL runs. In 2.x, MySQL drivers that format queries
+on the client (mysql2 before 3.17, mysql) expanded such an object into
+`column = value` pairs, so a value for one column could set others; PostgreSQL
+stored the JSON text and SQLite refused it. This also ships in 2.4.1.
+
+| 2.x | 3.0 |
+| --- | --- |
+| `{ type: 'none' }` holding an object | `{ type: 'object' }`, or a `storage.serialize` function |
+| `{ type: 'blob' }` given JSON | a `Buffer` or `Uint8Array` from server code |
+
+## `get` reads once
+
+A `get` on a resource with no `checkPermissions`, `checkDataPermissions`,
+`checkDataPermissionsGet`, `beforeData` or `beforeDataGet` hook (for the
+resource or global) reads its record in one statement, scope and row policy
+included, instead of a minimal read followed by a full read. Not found stays
+`not_found` and still answers before an invalid fieldset. Such a get no longer
+sets `context.minimalRecord`.
+
+| 2.x | 3.0 |
+| --- | --- |
+| `enrichRecord`, `finish` or `finishGet` reading `context.minimalRecord` | read `context.record`, or register a `checkPermissions` or `beforeDataGet` hook to keep the minimal read |
+
+## Write failure logs
+
+The `Error in METHOD method` entry that a failed write logs at error level now
+carries only the resource, the operation and error codes. In 2.x it also held
+the request's `inputRecord` (hidden fields redacted, every other value kept)
+and the formatted error, whose message and SQL can repeat the values being
+written, so logs collected payment state, ciphertext and other stored secrets.
+The error the call throws is unchanged and still carries its `cause`.
+
+| 2.x log field | 3.0 |
+| --- | --- |
+| `inputRecord` | removed; log what you need from the input yourself, in your own code |
+| `error`, `errorSummary` (message, stack, details) | `code`: the error's `code`, or its `name` when it has none |
+| `cleanupErrors: [{ phase, error }]` | `cleanupErrors: [{ phase, code }]` |
+| `method`, `scopeName`, `phase`, `backend`, `transactionOutcome` | unchanged |
+
+## HTTP error answers and logs
+
+A 500 answer for an error that is not one of the library's own `RestApi…Error`
+classes, such as a database error or a plain `Error` thrown by a hook, now has
+the detail `An unexpected error occurred`. In 2.x it carried the error's
+message, and a database error's message repeats the SQL with its values (Knex
+puts the statement and its bindings in it), so a duplicate key answered with
+the value that clashed. For a database error, the connector's `HTTP request
+error` or `Fastify request error` log entry now records only its `name` and
+`code`; PostgreSQL's `detail` and MySQL's `sql` repeat the values too.
+
+| 2.x | 3.0 |
+| --- | --- |
+| 500 `detail` is the error's message | the message for a `RestApi…Error`; otherwise `An unexpected error occurred` |
+| a hook throws `new Error(message)` to tell the client why | throw a `RestApiResourceError` or `RestApiValidationError`, or call `context.reject` in a transport hook |
+| the HTTP error log entry holds the database error (message, SQL, values) | `{ name, code }` of the database error |
+
 ## System contexts and `context.system`
 
 The autofilter and row-policy plugins now reserve `context.system`. A context
@@ -173,15 +289,27 @@ shared row on a `shared` filter. Row policies receive `system` alongside
 Filter definitions take `shared: true | false` (default false). A value other
 than a boolean, or `shared: true` on a field that is not `nullable`, fails when
 the resource is added. `getScopeConfig` reports `shared` for each filter.
+Outside a system context, shared rows refuse `patch`, `put`, `delete`,
+`postRelationship` and `deleteRelationship`, and a create may not leave the
+field NULL, also when an optional (`required: false`) filter does not resolve.
+
+## Values and links that read or write differently
+
+| 2.x | 3.0 |
+| --- | --- |
+| an `integer` field over a PostgreSQL BIGINT column reads `"4400"` | `4400`; a value past 2^53 stays a string |
+| an aggregate group, `min` or `max` of a BIGINT past 2^53 is a rounded number | the exact digits, as a string |
+| a `patch` or `put` of `accountId: null` by a writer who cannot see the account clears the link | keeps it; clear it in a system context |
+| `owner_type` and `owner_id` in attributes set a polymorphic link | refused; send `owner: { _type, id }` or `relationships.owner` |
 
 ## Custom storage plugins
 
-The storage helpers gain `dataScopedQuery`, `dataVisibleIds`,
+The storage helpers gain `dataScopedQuery`, `dataVisibleIds`, `dataIdsByKeys`,
 `dataInsertMany`, `dataUpdateByIds`, `dataDeleteByIds`, `dataLockByIds` and
 `dataClaim`. They are optional: a storage plugin without them keeps working,
 and the new methods refuse its resources with
 `RestApiBulkWriteUnsupportedError`. `RestApiKnexPlugin` implements all of them;
-`RestApiAnyapiKnexPlugin` implements none.
+`RestApiAnyapiKnexPlugin` implements only `dataScopedQuery`.
 
 ## Verify the port
 
@@ -189,6 +317,9 @@ and the new methods refuse its resources with
    `expectedVersions`, `defaultAtomic` and `operations:`, and port each call.
 2. Search for `system: true` in contexts.
 3. Run the application's tests. A many-row write on a resource with per-record
-   hooks fails with a message naming the hooks to give a batch form.
-4. Update HTTP clients of `/RESOURCE/bulk` to the new PATCH and DELETE bodies
+   hooks fails with `RestApiBulkWriteUnsupportedError`; its `details.hooks`
+   names the hooks to give a batch form.
+4. Search for `patchWhere` and `deleteWhere` calls without `filters`, and add
+   `all: true` where every visible row is meant.
+5. Update HTTP clients of `/RESOURCE/bulk` to the new PATCH and DELETE bodies
    and to error responses for failed batches.

package/lib/rest-api-errors.js

@@ -261,11 +261,11 @@ export class RestApiTemporalDataError extends RestApiError {
  * It never falls back to a per-row loop; the reason names what to change.
  */
 export class RestApiBulkWriteUnsupportedError extends RestApiError {
-  constructor ({ resourceType, operation, reason } = {}) {
+  constructor ({ resourceType, operation, reason, hooks } = {}) {
     super(`${operation} on '${resourceType}' is not supported: ${reason}`, 'REST_API_BULK_WRITE_UNSUPPORTED')
     this.type = 'rest_api_bulk_write_unsupported'
     this.statusCode = 422
-    this.details = { resourceType, operation, reason }
+    this.details = { resourceType, operation, reason, ...(hooks ? { hooks } : {}) }
   }
 
   toJSON () {
@@ -323,10 +323,11 @@ export class RestApiScopedQueryReadOnlyError extends RestApiError {
 
 /** queryAll stopped at the caller's rowLimit instead of reading an unbounded set. */
 export class RestApiRowLimitError extends RestApiError {
-  constructor ({ resourceType, rowLimit } = {}) {
-    super(`queryAll on '${resourceType}' matched more than ${rowLimit} rows; read them in a background job or narrow the filters`, 'REST_API_ROW_LIMIT')
+  constructor ({ resourceType, rowLimit, operation = 'queryAll' } = {}) {
+    const advice = operation === 'queryAll' ? 'read them in a background job or narrow the filters' : 'narrow the filters'
+    super(`${operation} on '${resourceType}' matched more than ${rowLimit} rows; ${advice}`, 'REST_API_ROW_LIMIT')
     this.type = 'rest_api_row_limit'
-    this.details = { resourceType, rowLimit }
+    this.details = { resourceType, rowLimit, operation }
   }
 
   toJSON () {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "json-rest-api",
-  "version": "3.0.0-rc.0",
+  "version": "3.0.1",
   "main": "index.js",
   "types": "index.d.ts",
   "type": "module",
@@ -142,7 +142,7 @@
   "devDependencies": {
     "@socket.io/redis-adapter": "^8.3.0",
     "@types/node": "24.0.14",
-    "better-sqlite3": "^11.10.0",
+    "better-sqlite3": "^13.0.3",
     "busboy": "^1.6.0",
     "eslint": "^9.37.0",
     "express": "^5.1.0",

package/plugins/core/connectors/lib/transport-http-helpers.js

@@ -23,8 +23,32 @@ export function mergeResponseHeaders (...sources) {
 export function getHttpDiagnosticError (error) {
   // Parser messages and stacks can quote raw input before a resource is known.
   // Keep the original error for hooks/responses, but log only this safe preview.
-  if (error?.type !== 'entity.parse.failed') return error
-  return { name: 'SyntaxError', message: 'Invalid JSON request body', type: 'entity.parse.failed', status: 400 }
+  if (error?.type === 'entity.parse.failed') {
+    return { name: 'SyntaxError', message: 'Invalid JSON request body', type: 'entity.parse.failed', status: 400 }
+  }
+  // A database error repeats the stored values: in the message Knex builds
+  // from the SQL and its bindings, in MySQL's sql and sqlMessage, and in
+  // PostgreSQL's detail. Log only which failure it was.
+  const failure = databaseFailure(error)
+  return failure ? { name: failure.name, code: failure.code } : error
+}
+
+/**
+ * The database driver error behind `error`, which may wrap it as its cause
+ * (a RestApiWriteError does).
+ * @param {unknown} error
+ */
+function databaseFailure (error) {
+  let current = error
+  for (let depth = 0; current && typeof current === 'object' && depth < 10; depth++) {
+    if (!isRestApiError(current)) {
+      const { sqlState, sqlMessage, errno, severity, code } = /** @type {any} */ (current)
+      // mysql2 (and socket) errors, pg errors, and better-sqlite3 or sqlite3 errors.
+      if (typeof sqlState === 'string' || typeof sqlMessage === 'string' || typeof errno === 'number' ||
+        typeof severity === 'string' || /^SQLITE_/.test(String(code ?? ''))) return current
+    }
+    current = /** @type {any} */ (current).cause
+  }
 }
 
 const JSON_API_WRITE_CONTENT_TYPES = [
@@ -115,11 +139,15 @@ export function mapRestApiErrorToHttp (error) {
   // Read hooks can reject with null or undefined.
   error ??= {}
   const statusHint = error.statusCode ?? error.status
+  // Only the library's own errors carry messages meant for clients; a write
+  // error wraps what failed. Other messages, a database's above all, can
+  // repeat SQL and stored values.
+  const origin = isRestApiError(error) && error instanceof RestApiWriteError ? error.cause : error
   let status = 500
   let errors = [{
     status: '500',
     title: 'Internal Server Error',
-    detail: error.message
+    detail: isRestApiError(origin) ? error.message : 'An unexpected error occurred'
   }]
 
   if (error?.validation) {

package/plugins/core/file-handling-plugin.js

@@ -226,7 +226,8 @@ export const FileHandlingPlugin = {
     // Uploads belong to one record each; many-row writes refuse rows that set file fields.
     const refuseFileFields = ({ context, scopeName }) => {
       const fileFields = getFileFields(scopeName)
-      const set = fileFields.filter(field => context.inputRecords.some(record => record?.data?.attributes?.[field.field] !== undefined))
+      const set = fileFields.filter(field => context.rawFields?.includes(field.field) ||
+        context.inputRecords.some(record => record?.data?.attributes?.[field.field] !== undefined))
       if (!set.length) return
       throw new RestApiBulkWriteUnsupportedError({
         resourceType: scopeName,

package/plugins/core/lib/anyapi/anyapi-registry.js

@@ -20,7 +20,8 @@ const DEFAULT_CACHE_SIZE = 100
 const EMPTY_RUNTIME = Object.freeze({ schema: {}, searchSchema: {}, callbacks: [] })
 
 // Errors from two servers inserting the same new resource at once.
-const WRITE_CONFLICT_CODES = new Set(['23505', '40001', '40P01', 'ER_DUP_ENTRY', 'ER_LOCK_DEADLOCK', 'SQLITE_BUSY', 'SQLITE_CONSTRAINT_UNIQUE'])
+// ER_CHECKREAD (1020): MariaDB with innodb_snapshot_isolation refuses a row changed since the snapshot.
+const WRITE_CONFLICT_CODES = new Set(['23505', '40001', '40P01', 'ER_DUP_ENTRY', 'ER_LOCK_DEADLOCK', 'ER_CHECKREAD', 'SQLITE_BUSY', 'SQLITE_CONSTRAINT_UNIQUE'])
 const creationRaces = new WeakSet()
 
 const resolveIdProperty = (schema = {}, explicitIdProperty = null) => {
@@ -113,7 +114,7 @@ const rowsSignature = (rows, columns) => JSON.stringify(rows.map(row => columns.
   return value ?? null
 })).sort((left, right) => String(left[0]).localeCompare(String(right[0]))))
 
-const isWriteConflict = error => WRITE_CONFLICT_CODES.has(error?.code) || [1062, 1213].includes(error?.errno)
+const isWriteConflict = error => WRITE_CONFLICT_CODES.has(error?.code) || [1020, 1062, 1213].includes(error?.errno)
 
 /**
  * Canonical resource metadata. One registry can serve every tenant API that

package/plugins/core/lib/anyapi/canonical-link-store.js

@@ -432,7 +432,10 @@ export function createCanonicalLinkStore ({ api, getDescriptor, getScopeStorageA
     resolveId: row => row.relationship === options.relationshipKey && row.left_resource === options.descriptor.resource && normalizeId(row.left_id) === options.leftId ? row.right_id : row.left_id
   })
 
-  const invalidateDeletedLinkTargets = async (scopeName, context) => {
+  // Advances the versions of parents linked to a record about to be deleted.
+  // When there are such parents, `lockRow` first locks the record's row against
+  // link writers, and answers not found when it has gone.
+  const invalidateDeletedLinkTargets = async (scopeName, context, lockRow) => {
     const relationships = []
     for (const [parentType, parent] of Object.entries(api.resources)) {
       if (!parent.vars.schemaInfo?.versionField) continue
@@ -444,7 +447,7 @@ export function createCanonicalLinkStore ({ api, getDescriptor, getScopeStorageA
     }
     if (!relationships.length) return
     assertVersionedRelationshipTransaction(api, relationships[0].inverse, context.transaction)
-    await lockRelationshipTargets(api, context.transaction, [{ type: scopeName, id: context.id }], { exclusive: true })
+    await lockRow()
     const id = normalizeId(context.id)
     for (const { inverse, info } of relationships) {
       const query = buildLinkMutationQuery({ ...info, db: context.transaction, allParents: true }, (query, column) => query.where(column, id))