json-rest-api 3.0.0-rc.0 → 3.0.0

package/README.md

@@ -75,8 +75,8 @@ app.listen(3000, () => {
 
 Version 3 adds many-row writes (`postMany`, `patchMany`, `patchWhereIds`,
 `patchWhere`, `deleteMany`, `deleteWhere`), scoped reads (`scopedQuery`,
-`queryAll`), row locks (`lockByIds`, `claim`), lock versions, shared rows and
-system contexts, and replaces the 2.x bulk methods. The
+`queryAll`), row locks (`lockByIds`, `lockWhere`, `claim`), lock versions, shared
+rows and system contexts, and replaces the 2.x bulk methods. The
 [v3 migration guide](docs/GUIDE/35-migrating-to-v3.md) lists the breaking changes
 with old → new tables; the [v2 migration guide](docs/GUIDE/33-migrating-to-v2.md)
 covers upgrades from 1.x.

package/docs/API.md

@@ -86,7 +86,7 @@ a separate context for each enlisted operation. See [transaction support](#trans
    - [scopedQuery - Scoped Read-Only Builder](#scopedquery---scoped-read-only-builder)
    - [queryAll - Every Matching Record](#queryall---every-matching-record)
    - [Many-Row Writes](#many-row-writes)
-   - [lockByIds and claim - Row Locks](#lockbyids-and-claim---row-locks)
+   - [lockByIds, lockWhere and claim - Row Locks](#lockbyids-lockwhere-and-claim---row-locks)
    - [System Contexts](#system-contexts)
 4. [Relationship Methods](#relationship-methods)
    - [getRelated - Retrieve Related Resources](#getrelated---retrieve-related-resources)
@@ -1177,6 +1177,11 @@ foreign keys. Fields that are `hidden`, virtual, computed, or that have a getter
 the ones clients read. The exception is a `date` or `dateTime` field with
 `storage.serialize`: it may be bucketed and take `min` and `max`, whose results
 are read back as that field's values. `normallyHidden` fields are allowed.
+In a [system context](GUIDE/34-scoped-reads-and-locks.md#system-contexts), a
+hidden ownership (autofilter) field may also be grouped and take
+`countDistinct`, so a job can find which owners have rows; other callers are
+refused it, since one whose optional owner filter does not resolve would see
+every owner.
 
 #### Conditional measures
 
@@ -1273,13 +1278,16 @@ const { query } = await api.resources.entries.scopedQuery(
   { filters, alias, purpose, transaction }, context)
 ```
 
-Resolves to `{ query }`, a Knex builder on the resource's table with ownership
-filters, row policy and the search `filters` applied. It is wrapped because a
-builder is thenable. `purpose` is `'report'` (default; includes shared rows),
-`'bulk-write'`, `'lock'` or `'claim'`; row policies receive it as
+Resolves to `{ query }`, a Knex builder over a derived table of the resource's
+rows with ownership filters, row policy and the search `filters` applied, so
+conditions the caller adds (`orWhere` included) only narrow them. It is wrapped
+because a builder is thenable. `purpose` is `'report'` (default; includes
+shared rows), `'bulk-write'`, `'lock'` or `'claim'`; row policies receive it as
 `queryPurpose`. The caller needs `query` permission. The builder refuses
-writes, locking reads, table changes and clearing predicates, on clones too
-(`RestApiScopedQueryReadOnlyError`); joined tables are not scoped.
+writes, locking reads, table changes, clearing predicates, set operations,
+common table expressions and access to its Knex client, on clones too
+(`RestApiScopedQueryReadOnlyError`); joined tables are not scoped. It guards
+against mistakes, not hostile code.
 
 ### queryAll - Every Matching Record
 
@@ -1299,20 +1307,20 @@ it throws `RestApiRowLimitError`. Works on both storages.
 await resource.postMany({ data | document, onConflict, returning, format, queryParams, chunkSize, transaction }, context)
 await resource.patchMany({ data | document, onlyIf, missing, returning, format, queryParams, chunkSize, transaction }, context)
 await resource.patchWhereIds({ ids, data | document, onlyIf, missing, returning, format, queryParams, chunkSize, transaction }, context)
-await resource.patchWhere({ filters, data | document, onlyIf, returning, format, queryParams, chunkSize, transaction }, context)
+await resource.patchWhere({ filters | all: true, data | document, onlyIf, returning, format, queryParams, chunkSize, transaction }, context)
 await resource.deleteMany({ ids, missing, returning, chunkSize, transaction }, context)
-await resource.deleteWhere({ filters, returning, chunkSize, transaction }, context)
+await resource.deleteWhere({ filters | all: true, returning, chunkSize, transaction }, context)
 ```
 
 | Parameter | Description |
 | --- | --- |
 | `data` / `document` | `postMany`, `patchMany`: an array of plain records, or `{ data: [resource objects] }`; `patchMany` rows carry `id`. `patchWhereIds`, `patchWhere`: one record of changes. |
-| `ids` / `filters` | The records to change: listed ids, or search filters validated as `query` validates them. |
-| `returning` | `'none'` (default), `'minimal'` (`{ data: [{ type, id }] }`) or `'full'` (`{ data: [records] }`, not for deletes). |
+| `ids` / `filters` / `all` | The records to change: listed ids, or non-empty search filters validated as `query` validates them, or `all: true` for every row the caller can see. |
+| `returning` | `'none'` (default), `'minimal'` (`{ data: [{ type, id }] }`), `'full'` (`{ data: [records] }`, not for deletes) or `'count'` (`{ meta: { count } }`, the number of rows written, with no row data). |
 | `format`, `queryParams` | Output format, `fields` and `include` of full records. |
-| `onConflict` | Upsert: `{ keys, action: 'ignore' \| 'merge', merge? }`. `keys` must include every ownership field; `merge` is refused under a row policy. |
+| `onConflict` | Upsert: `{ keys, action: 'ignore' \| 'merge', merge? }`. `keys` must include every ownership field, and a unique index must cover them; rows may not carry an `id` or repeat keys. On a resource with no ownership fields and no row policy, `keys: ['id']` (or the `idProperty`) keys the upsert on the id, and every row carries one; keys compare exactly (a match only by collation fails with 409); only a match on the keys counts, and a clash on another unique index fails the batch. `merge` updates only the fields each row supplies (never `defaultTo` values), runs the patch checks (`checkPermissions` with `method: 'patch'`) on the records it updates and is refused under a row policy. |
 | `onlyIf` | Compare-and-set: only rows whose stored values equal these change and are returned. |
-| `missing` | `'throw'` (default; `RestApiRecordsNotFoundError`, 404) or `'skip'` for ids the caller cannot see. |
+| `missing` | Id lists only: `'throw'` (default; `RestApiRecordsNotFoundError`, 404) or `'skip'` for ids the caller cannot see. `patchWhere` and `deleteWhere` refuse it: they write the rows their selection finds, with the scope and filters inside each write, and return only the rows written. |
 | `chunkSize` | Rows per statement (default 500), lowered to the database's parameter limit. |
 | `transaction` | A transaction from `api.transaction` to join; otherwise the call owns one. |
 
@@ -1320,9 +1328,12 @@ Each call is all-or-nothing. Every row is validated, stamped, passed through
 setters and authorized with `checkPermissions` (`method` `post`, `patch` or
 `delete`, `originalContext.bulkIndex` the row's position); belongs-to targets
 are checked with one read per related type; then one statement runs per chunk.
-Patches refuse the id, ownership fields and the `lockVersionField`, and accept
-`knex.raw()` values for stored, non-relationship fields. Deletes remove the
-records' pivot rows. Batch hooks run once per call:
+Patches refuse the id, ownership fields (also through a polymorphic
+relationship) and the `lockVersionField`, and accept real `knex.raw()` values
+for stored, non-relationship fields. The UPDATE, DELETE and lock statements
+repeat the caller's ownership conditions, so a row that moved to another owner
+after the visibility read is left alone and counts as missing. Deletes remove
+the records' pivot rows. Batch hooks run once per call:
 `beforeProcessingPostMany`, `beforeDataCallPostMany`, `afterDataCallPostMany`,
 the `PatchMany` trio and `beforeDataCallDeleteMany` / `afterDataCallDeleteMany`.
 
@@ -1333,23 +1344,37 @@ many-to-many or reverse relationships, file fields, or positions.
 `BulkOperationsPlugin` exposes `postMany`, `patchMany` and `deleteMany` as
 `POST`, `PATCH` and `DELETE /RESOURCE/bulk`.
 
-### lockByIds and claim - Row Locks
+### lockByIds, lockWhere and claim - Row Locks
 
 ```javascript
 await api.transaction(async transaction => {
   const { data: locked } = await resource.lockByIds({ ids, transaction, missing, format, queryParams: { fields } }, context)
+  const { data: matching } = await resource.lockWhere({ filters, transaction, format, queryParams: { fields } }, context)
   const { data: taken } = await resource.claim({ filters, sort, limit, transaction, format, queryParams: { fields } }, context)
 })
 ```
 
-Both need a transaction (`RestApiLockOutsideTransactionError` otherwise) and
-return the records as the locking read saw them. `lockByIds` locks the visible
-rows among `ids` in id order (`FOR UPDATE`); hidden or missing ids throw
-`RestApiRecordsNotFoundError` unless `missing: 'skip'`. `claim` takes up to
-`limit` (default 1, at most 1,000) matching rows that no other transaction holds
-(`SKIP LOCKED`), in `sort` order (stored fields, default `['id']`); a row policy
-that joins tables is refused. On SQLite, which has one writer, both are scoped
-reads.
+All three need a transaction (`RestApiLockOutsideTransactionError` otherwise)
+and return the records as the locking read saw them, through the read hooks of
+`query` (`enrichRecord`, `finish`, `finishQuery`) and `checkDataPermissions`
+for each row. The locking read is the first statement and applies the scope
+itself, so plain reads later in the transaction see what a previous lock holder
+committed. `lockByIds` locks the visible rows among `ids` in id order
+(`FOR UPDATE`), sorting lists longer than its 1,000-id statements first; hidden
+or missing ids throw `RestApiRecordsNotFoundError` unless `missing: 'skip'`. On
+MySQL and MariaDB it locks every listed id, hidden ones too, until commit, so
+pass ids from a scoped read, never a raw request list. `lockWhere` locks the
+visible rows the filters select in id order, waiting for other transactions and
+including rows that start to match meanwhile; more than 1,000 throw
+`RestApiRowLimitError`. On MySQL and MariaDB it has no `ORDER BY` and forces an
+index that starts with the whole ownership columns (or, without ownership
+conditions, a filtered column), and throws `RestApiBulkWriteUnsupportedError`
+when the table has none.
+`claim` takes up to `limit` (default 1, at most 1,000) matching rows that no
+other transaction holds (`SKIP LOCKED`), in `sort` order (stored fields, default
+`['id']`). A row policy that joins tables is refused for all three. On SQLite,
+which has one writer, they take the database's write lock with a write that
+changes nothing, then read.
 
 A resource option `lockVersionField: 'lock_version'` names a stored integer
 that every update (single, many-row and upsert merge) sets to `column + 1`.
@@ -1992,8 +2017,9 @@ const result = await api.resources.articles.query({
 ```
 
 Pagination metadata is under `meta.pagination`. With counts disabled, it includes
-`page` and `pageSize`, without
-promising totals or `hasMore`. The library supplies pagination links appropriate
+`page`, `pageSize` and `hasMore`, without totals: the page reads one extra row
+instead of running a count query, so it is one statement and links to `first`,
+`prev` and `next` but not `last`. The library supplies pagination links appropriate
 to the available count/boundary information.
 
 #### Cursor Pagination
@@ -2377,6 +2403,13 @@ For example, a resource not-found error maps to:
 | `REST_API_SYSTEM_CONTEXT_FORGED` | 500 |
 | Unexpected or temporal output failure | 500 |
 
+A 500 answer carries the error's message only for the library's own
+`RestApi…Error` classes. Any other error, a database error or a plain `Error`
+thrown by a hook, answers with the detail `An unexpected error occurred`: a
+database error's message repeats the SQL and its values. The connector's
+`HTTP request error` or `Fastify request error` log entry records only the
+`name` and `code` of a database error.
+
 Validation violations produce individual HTTP errors with their message and
 field location; generic validation errors retain their available details.
 Relationship validation/access failures follow their actual typed classification,

package/docs/GUIDE/07-polymorphic-relationships.md

@@ -98,8 +98,10 @@ const authorReviewDocument = await api.resources.reviews.post({
 
 Plain polymorphic input uses `{ _type, id }` under the relationship alias.
 JSON:API uses `{ type, id }` under `data.relationships.reviewable.data`.
-Do not write `reviewable_type` or `reviewable_id` as public attributes. Those
-logical fields describe storage; the public relationship represents them together.
+Writes refuse `reviewable_type` and `reviewable_id` in attributes (rule
+`foreign_key_in_attributes`), as they refuse a belongs-to foreign key: those
+logical fields describe storage, and the public relationship sets them together,
+after checking that the caller can see the target.
 Plain output uses `_type` to identify the related resource type as well.
 
 ## Include either direction

package/docs/GUIDE/09-pagination-and-sorting.md

@@ -65,8 +65,10 @@ enabled, metadata is:
 ```
 
 Page 10 returns an empty array. `enablePaginationCounts: false` suppresses the
-count query and count-derived metadata; do not assume every collection response
-contains totals. Every mode respects `queryMaxLimit`.
+count query and count-derived metadata (`total`, `pageCount` and the `last`
+link); do not assume every collection response contains totals. The page then
+reads one extra row, so `hasMore` and the `next` link remain, and a numbered
+page costs one statement. Every mode respects `queryMaxLimit`.
 
 Server code that reads every row, for example an export or a nightly job, can
 raise the cap for one call with `pageSizeLimit`, a parameter beside

package/docs/GUIDE/13-hooks-and-lifecycle.md

@@ -393,7 +393,12 @@ a handler to these resource write stages.
 ## Read order and output enrichment
 
 GET authorizes through `checkPermissions`, then runs `beforeData` and
-`beforeDataGet`. After fetching, it runs `checkDataPermissions`,
+`beforeDataGet`. These see `context.minimalRecord`, the stored row read before
+the full read. A resource with none of these three hooks and no
+`checkDataPermissions` or `checkDataPermissionsGet` hook (registered for it or
+globally) reads its row once, with its scope and row policy in that read, and
+leaves `minimalRecord` unset; a missing or hidden row is still `not_found`, and
+answers before an invalid fieldset does. After fetching, it runs `checkDataPermissions`,
 `checkDataPermissionsGet` and `enrichRecord`. Field getters and computed output
 run during attribute enrichment, followed by `enrichAttributes`.
 `enrichRecordWithRelationships`, `finish` and `finishGet` follow. Final

package/docs/GUIDE/16-autofiltering.md

@@ -300,7 +300,7 @@ await api.addResource('categories', {
 
 - Reads (`query`, `get`, includes, `aggregate`, `scopedQuery`) see the owner's
   rows and the shared rows.
-- Many-row writes, `lockByIds`, `claim` and `scopedQuery` with purpose
+- Many-row writes, `lockByIds`, `lockWhere`, `claim` and `scopedQuery` with purpose
   `'bulk-write'`, `'lock'` or `'claim'` see only the owner's rows.
 - Single `patch`, `put` and `delete` of a shared row fail as `forbidden`, and so
   does a create whose owner value resolves to NULL.

package/docs/GUIDE/17-row-policies.md

@@ -296,7 +296,7 @@ Policies are reused for the resource selections that define externally visible d
 | `relationship-parent` | The parent of a relationship route such as `getRelated`. |
 | `report` | A `scopedQuery()` builder with the default purpose. |
 | `bulk-write` | The visible ids a many-row patch or delete may change, and `scopedQuery({ purpose: 'bulk-write' })`. |
-| `lock` | The visible ids `lockByIds()` locks. |
+| `lock` | The rows `lockByIds()` and `lockWhere()` lock, in the locking read itself. A policy that joins other tables is refused here, as for `claim`. |
 | `claim` | The candidate rows `claim()` takes. A policy that joins other tables is refused here, because the database would lock the joined rows too. |
 | `unspecified` | A custom call to the lower-level query-filtering seam did not provide a purpose. |
 
@@ -316,11 +316,13 @@ and include limits in their operation/resource settings. Visibility hooks add
 row predicates and retain the query's selected ID and mandatory constraints.
 Their invocation count is not a per-request or per-record contract.
 
-Regular and polymorphic belongs-to includes also load at most 100 target IDs
-per query, with target permission and filtering hooks applied to every batch
-under `queryPurpose: 'include'`. SQL projection callbacks run for each query.
-Nested collection includes run after all target batches have been collected,
-so a global collection limit is not restarted for each batch.
+Regular and polymorphic belongs-to includes load each target type's records
+in one query, however many there are, with target permission and filtering
+hooks applied under `queryPurpose: 'include'`. A long id list is bound as one
+parameter on PostgreSQL and SQLite, so it stays within their parameter limits;
+MySQL and MariaDB have none. SQL projection callbacks run for each query.
+Nested collection includes run after all targets have been collected, so a
+global collection limit applies to the complete set.
 
 Relationship validation uses the same batch bound. Its query context has no
 individual `id` and no parent client filters. Retain the selected ID and scope

package/docs/GUIDE/21-schema-and-migrations.md

@@ -596,7 +596,13 @@ Current live introspection support:
 
 - SQLite
 - PostgreSQL via `pg` (verified on PostgreSQL 16)
-- MySQL via `mysql2` (verified on MySQL 8; this does not establish MariaDB coverage)
+- MySQL via `mysql2` (verified on MySQL 8)
+- MariaDB via `mysql2` (verified on MariaDB 11.4)
+
+MariaDB reports column defaults as SQL (a quoted literal, the word `NULL` for no
+default) and stores a `json` column as `longtext` with a `json_valid` column
+check; the snapshot reads both as MySQL's forms, so the JSON check is part of
+the column type and not a check constraint of its own.
 
 PostgreSQL inspection reads the current schema's
 [columns](https://www.postgresql.org/docs/16/infoschema-columns.html),

package/docs/GUIDE/27-many-row-writes.md

@@ -13,9 +13,9 @@ Every table-storage resource has six methods that write many rows at once:
 | `postMany({ data })` | one new record per entry |
 | `patchMany({ data: [{ id, ... }] })` | each entry's own changes to the record with that id |
 | `patchWhereIds({ ids, data })` | the same changes to every listed record |
-| `patchWhere({ filters, data })` | the same changes to every record the filters select |
+| `patchWhere({ filters, data })` | the same changes to every record the filters select (or `all: true`) |
 | `deleteMany({ ids })` | the listed records |
-| `deleteWhere({ filters })` | every record the filters select |
+| `deleteWhere({ filters })` | every record the filters select (or `all: true`) |
 
 Each call is all-or-nothing. Every row still goes through validation,
 ownership stamping (autofilter), setters, relationship checks and
@@ -61,9 +61,11 @@ console.log(created.data, restocked.data)
 ```
 
 Rows come back in input order. Many-row writes return nothing unless you ask:
-`returning: 'minimal'` gives identifiers and `returning: 'full'` the records,
-read back in the write's transaction. The second call is an upsert: Alpha's
-ISBN already exists, so its stock becomes 9, and Gamma is inserted.
+`returning: 'minimal'` gives identifiers, `returning: 'full'` the records,
+read back in the write's transaction, and `returning: 'count'` only
+`{ meta: { count } }`, the number of rows written, without reading any back.
+The second call is an upsert: Alpha's ISBN already exists, so its stock
+becomes 9, and Gamma is inserted.
 
 ```javascript
 const [alpha, beta] = created.data
@@ -109,10 +111,10 @@ await api.resources.books.postMany({
 ```
 
 Rows may set belongs-to relationships. The library checks every target the
-batch names with one scoped read per related type and locks the targets once
-for the batch. Rows that set many-to-many or reverse (has-many) relationships
-are refused; change those links with single writes or the relationship
-methods.
+batch names with one scoped read per related type, and locks them once for the
+batch with one statement per type, however many rows name them. Rows that set
+many-to-many or reverse (has-many) relationships are refused; change those
+links with single writes or the relationship methods.
 
 `patchMany`, `patchWhereIds` and `patchWhere` refuse:
 
@@ -120,6 +122,10 @@ methods.
 - autofilter (ownership) fields and the relationships that set them;
 - the resource's `lockVersionField`, which every update advances by itself.
 
+A row of a resource with a lock version may change no field at all: its UPDATE
+then only advances the version, which tells other writers' `onlyIf` the
+record has changed.
+
 They accept a `knex.raw(...)` value for a stored, non-relationship field, so a
 counter can move inside the statement:
 
@@ -130,6 +136,12 @@ await api.resources.books.patchWhereIds({
 })
 ```
 
+Only a real Knex raw value counts. An object that merely carries an
+`isRawInstance` flag, as JSON input can, is refused with a validation error, so
+the HTTP routes never pass SQL through. Batch hooks find the fields that raw
+values set in `context.rawFields`, and the file-field and positioning refusals
+below apply to them as well.
+
 Every row also goes through the resource's setters, and creates through
 `defaultTo` values and autofilter stamping, as single writes do. Rows may omit
 different fields. On SQLite, a field that some rows set and others omit is
@@ -140,13 +152,14 @@ does not declare with `defaultTo` does not apply to them.
 
 | Option | Methods | Default | Meaning |
 | --- | --- | --- | --- |
-| `returning` | all | `'none'` | `'minimal'` returns `{ data: [{ type, id }] }`; `'full'` (not for deletes) returns the records |
+| `returning` | all | `'none'` | `'minimal'` returns `{ data: [{ type, id }] }`; `'full'` (not for deletes) returns the records; `'count'` returns `{ meta: { count } }` |
 | `format` | writes | resource default | Output format of `'full'` records |
 | `queryParams` | writes | | `fields` and `include` of `'full'` records |
 | `transaction` | all | | A transaction from `api.transaction` to join |
 | `chunkSize` | all | 500 | Rows per statement, lowered further to the database's parameter limit |
 | `missing` | `patchMany`, `patchWhereIds`, `deleteMany` | `'throw'` | `'skip'` ignores ids the caller cannot see |
 | `onlyIf` | patches | | Compare-and-set: change only rows whose stored values equal these |
+| `all` | `patchWhere`, `deleteWhere` | | `true` acts on every row the caller can see; needed when there are no `filters` |
 | `onConflict` | `postMany` | | Upsert, below |
 
 Ids that do not exist or that the caller cannot see (another owner's rows,
@@ -155,6 +168,95 @@ rows a row policy hides) throw `RestApiRecordsNotFoundError` (subtype
 count, so it does not reveal which ids exist. With `missing: 'skip'` they are
 left out. A batch may not name the same id twice.
 
+With `missing: 'skip'`, `patchMany`, `patchWhereIds` and `deleteMany` read
+nothing first when the resource has no `checkPermissions` hook and no
+`beforeDataCallPatchMany` or `beforeDataCallDeleteMany` hook (those need the
+visible rows before the write), except on MySQL and MariaDB. The UPDATE or
+DELETE carries the caller's scope, row policy included, as a subquery, and its
+changed rows are the answer, returned with `RETURNING` on PostgreSQL and
+SQLite. MySQL and MariaDB keep the visibility read for ids from the caller:
+InnoDB would lock every listed row before checking the scope, rows the caller
+cannot see included, and hold those locks until commit. After that read, a
+statement of one row there runs the UPDATE with its ownership conditions and
+`onlyIf`, and its row count is the answer. A longer statement locks the rows
+that still match and then updates exactly those, and so does a one-row
+statement on a connection that disables the drivers' default `FOUND_ROWS`
+flag, where MySQL counts the rows an UPDATE changed rather than those it
+matched.
+
+`patchWhere` and `deleteWhere` are set operations. They read the ids of the
+rows their selection finds, then write those rows with the caller's scope and
+the filters inside every UPDATE or DELETE (on MySQL and MariaDB inside a locking
+read of the rows, followed by the write of exactly those). A row that another
+transaction deletes, moves out of the caller's scope or changes so that it no
+longer matches in the meantime is left out. The result lists the rows actually
+written, and the methods never throw `RestApiRecordsNotFoundError`; they take no
+`missing` option.
+
+The filters are checked on each row as it is once locked, so a selection write
+is a compare-and-set on them: of two transactions whose filters matched the
+same row, the one that waits for the other's lock finds the row the other
+committed, and leaves it out if it no longer matches. PostgreSQL locks the rows
+first, so the statement that writes them starts after the wait; MySQL and
+MariaDB check the scope and the filters in the locking read itself, which
+therefore also locks the rows it joins from other tables. With MariaDB's
+`innodb_snapshot_isolation` on, the waiting transaction fails instead ("Record
+has changed since last read", errno 1020); retry it. SQLite has one writer:
+the second transaction fails with `SQLITE_BUSY` rather than write over the
+first, or, if it starts after the first commits, finds the row the first
+committed.
+
+A filter on another table's field (a `searchSchema` field whose `actualField`
+names another table) is read through a derived table. PostgreSQL reads it
+again after the wait, so it too is a compare-and-set; InnoDB would read it from
+the snapshot without locking it, so on MySQL and MariaDB `patchWhere` and
+`deleteWhere` refuse filters whose scope joins a derived table (rule
+`selection_cross_table`), whatever rows they match. Filter on the row's own
+columns instead, for example on the link itself after locking its targets with
+`lockByIds` on their resource. A custom `applyFilter` that reads another table
+through its own subquery is not detected: MySQL and MariaDB read that subquery
+from the snapshot too, so it is no compare-and-set there.
+
+The UPDATE, DELETE and lock statements repeat the caller's ownership
+(autofilter) conditions as plain column predicates. A row that another
+transaction moves to a different owner between the visibility read and the
+write is therefore neither written nor locked, and counts as missing:
+`missing: 'throw'` rolls the batch back, `'skip'` leaves the row out of the
+result, and with `onlyIf` it is reported unchanged. Single `patch`, `put` and
+`delete`, on table and AnyAPI storage, do the same with their one scoped read:
+the UPDATE or DELETE carries the owner that read found, and a row that has gone
+or moved to another owner since answers `not_found`. The statement's own row
+count tells. A DELETE counts the rows it removed, and an UPDATE the rows it
+matched on PostgreSQL, SQLite and default MySQL connections. On a MySQL or
+MariaDB connection without `FOUND_ROWS`, an UPDATE that changes nothing counts
+zero, so a zero count there is checked again with a locking read of the row.
+Under READ COMMITTED that re-check leaves a window: a row deleted and created
+again with the same id and owner between the UPDATE and the re-check passes
+it, and the write is reported done although nothing was written. REPEATABLE
+READ, InnoDB's default, closes it: the UPDATE's gap lock holds the new row off
+until commit. A proxy that does not pass the `FOUND_ROWS` flag on to its
+backend connections makes a no-op `patch` of a resource without a lock version
+answer `not_found`.
+
+Row policies are checked by the visibility read only, for single and
+many-row writes alike: a row that another transaction moves out of the policy
+(for example into a group the caller cannot see) between the read and the
+write is still written. The ownership conditions close that window for the
+owner. The policy would need a subquery on the written table, which MySQL
+refuses in an UPDATE or DELETE, so no database checks it there. To keep a
+policy-relevant change from slipping between the read and the write, lock the
+row in the writing transaction first (`lockByIds`, which helps only for a
+policy on the row's own columns: [it refuses](17-row-policies.md) a policy
+that joins other tables), or give a many-row write an `onlyIf` on the policy's
+fields. Without a
+visibility read (`missing: 'skip'`, above) the policy is checked by the
+write's subquery, which reads as of the statement's start.
+
+`patchWhere` and `deleteWhere` need non-empty `filters`. To change or delete
+every row the caller can see, pass `all: true` instead (not both); in a
+[system context](34-scoped-reads-and-locks.md#system-contexts) that is the whole
+table.
+
 `onlyIf` makes a patch conditional. Only the rows whose stored values still
 equal the given ones change, and with `returning: 'minimal'` or `'full'` the
 result lists only those rows. With a
@@ -175,21 +277,81 @@ if (changed.data.length === 0) {
 ## Upserts
 
 `onConflict: { keys, action, merge }` turns `postMany` into an upsert.
-`keys` are the stored fields of a unique index; each row must give them values.
-With `action: 'ignore'`, rows that conflict are skipped and only the inserted
-rows are returned. With `action: 'merge'`, the conflicting record is updated:
-the fields in `merge`, or by default every stored field the rows supply except
-the keys, the id, ownership fields and the lock version.
+`keys` are the stored fields of a unique index. Each row must give them values,
+no two rows may repeat them, and rows may not carry an `id`: the keys identify
+the record. On a resource with no autofilter (ownership) fields and no row
+policy, `keys: ['id']` makes the id itself the key (the `idProperty` name, such
+as `email` for a natural key, means the same); every row then carries its id,
+and the primary key is the unique index. There is no other owner whose row the
+id could match, so this is the atomic create-if-missing for such tables.
+With `action: 'ignore'`, rows that match an existing record are skipped and
+only the inserted rows are returned. With `action: 'merge'`, the matching
+record is updated: the fields in `merge` that the row supplies, or by
+default every stored field the rows supply except the keys, the id, ownership
+fields and the lock version. A row supplies the fields the caller sent, its
+belongs-to links and the fields a setter adds; schema defaults (`defaultTo`)
+apply only to the rows merge inserts, never to the records it updates.
+
+Keys compare exactly. A stored value is first read as its field reads it, so
+`true` matches a boolean stored as 1 and `1.5` a decimal stored as `1.50`, but
+strings compare as they are, without the column's collation. When the database
+finds a record that matches the keys only by its own comparison, for example
+`'Urgent'` against `'urgent'` under MySQL's case-insensitive collations, the
+batch fails with a `conflict` error (409) instead of guessing.
 
 Upserts cannot reach another owner's rows:
 
 - `keys` must include every autofilter (ownership) field of the resource. The
-  unique index should include them too.
-- `merge` is refused on a resource with a row policy, because the database
-  cannot apply the policy to the row it merges into. Insert with `ignore`, then
-  `patchMany` the rows the caller may change.
-- On MySQL and MariaDB, a row that conflicts on a different unique index fails
-  the batch instead of updating that row.
+  unique index should include them too. The id is a key only on a resource
+  with no ownership fields and no row policy.
+- `merge` is refused on a resource with a row policy. Insert with `ignore`,
+  then `patchMany` the rows the caller may change.
+- Only a match on the conflict keys counts. A row that clashes with an existing
+  record on any other unique index, the primary key included, fails the batch.
+- `merge`, and `ignore` on MySQL and MariaDB, need a unique index whose
+  columns are all among the keys, and refuse the batch
+  (`RestApiBulkWriteUnsupportedError`) without one. Without it the keys could
+  match several records, and the locking read would examine, and on MySQL
+  lock, every row of the table. (`ignore` on PostgreSQL and SQLite needs a
+  unique index on exactly the keys, as `ON CONFLICT` does.) The first upsert
+  on a table reads its indexes once.
+
+`merge` reads the records that match the keys under a row lock, inserts the
+other rows with a plain INSERT and updates the matched records by id, a fixed
+number of statements per chunk. The records it updates go through a patch's
+checks as well as a create's: `checkPermissions` runs with `method: 'patch'`
+and the matched record as `minimalRecord`, the resource is refused if a
+per-record patch hook without `{ batch: true }` runs for it, and a key that
+matches a record the caller cannot see fails the batch as `forbidden`.
+
+With `returning: 'count'` an upsert counts the rows it inserted, and with
+`merge` the records it updated as well.
+
+`ignore` runs `INSERT … ON CONFLICT DO NOTHING` on PostgreSQL and SQLite. On
+MySQL and MariaDB, whose `ON DUPLICATE KEY` fires on every unique index, it
+reads the matching keys under a row lock and inserts the rest. With `merge`,
+and with `ignore` on MySQL and MariaDB, two transactions that upsert the same
+new key at the same time can fail with a unique-key error (PostgreSQL 23505)
+or a deadlock (MySQL 1213); retry the batch. InnoDB keeps the locks of every
+row a locking read examined until the transaction ends, so keep upserting
+transactions short.
+
+A single `put` creates a record at the id the caller chose, and is the one
+write that tells an id taken by another owner or hidden from the caller
+apart from a free one: it
+answers `not_found` for the first and creates the second. Where ids must not
+leak between owners, let the server choose them: create with `post` or
+`postMany` (or upsert on keys that include the owner) and keep `put` for
+records the caller already has.
+
+`merge` on every database, and `ignore` on MySQL and MariaDB, find the rows
+that match the conflict keys with a locking read of the bare table, before the
+caller's scope is checked. A matching row the caller cannot see, such as one a
+`knexQueryFiltering` hook hides, is therefore locked until commit, and waited
+for while another transaction holds it, as `lockByIds` does with a hidden id
+(see [scoped reads and locks](34-scoped-reads-and-locks.md)). `merge` then fails
+the batch as `forbidden`, and `ignore` skips the row. With `keys: ['id']` that
+read locks the listed ids by primary key.
 
 ## Transactions
 
@@ -219,12 +381,15 @@ hooks:
 | Method | Batch hooks (in order) | Context |
 | --- | --- | --- |
 | `postMany` | `beforeProcessingPostMany`, `beforeDataCallPostMany`, `afterDataCallPostMany` | `inputRecords`; `ids` after the insert |
-| `patchMany`, `patchWhereIds`, `patchWhere` | `beforeProcessingPatchMany`, `beforeDataCallPatchMany`, `afterDataCallPatchMany` | `inputRecords`, `ids` |
+| `patchMany`, `patchWhereIds`, `patchWhere` | `beforeProcessingPatchMany`, `beforeDataCallPatchMany`, `afterDataCallPatchMany` | `inputRecords`, `rawFields`, `ids` |
 | `deleteMany`, `deleteWhere` | `beforeDataCallDeleteMany`, `afterDataCallDeleteMany` | `ids` |
 
-`afterCommit` and `afterRollback` run once, with `context.method` set to the
-many-row method. `checkPermissions` still runs for every row, with `method`
-`'post'`, `'patch'` or `'delete'` and the row's position as
+In the `afterDataCall…Many` hooks, `ids` lists only the rows the write
+changed, and for patches `inputRecords` lists their inputs in the same order,
+so a row that `missing: 'skip'`, `onlyIf` or a selection left out appears in
+neither. `afterCommit` and `afterRollback` run once, with `context.method` set
+to the many-row method. `checkPermissions` still runs for every row, with
+`method` `'post'`, `'patch'` or `'delete'` and the row's position as
 `originalContext.bulkIndex`.
 
 A resource whose per-record write hooks would be skipped is refused rather than
@@ -246,10 +411,12 @@ HTTP 422) before writing when:
 - the resource has a `versionField`, or its writes advance another resource's
   version (use single writes with `expectedVersion`, or a `lockVersionField`
   with `onlyIf`);
-- a per-record write hook without `{ batch: true }` runs for the resource;
+- a per-record write hook without `{ batch: true }` runs for the resource
+  (the error's `details.hooks` names them; HTTP responses leave details out);
 - rows set many-to-many or reverse relationships;
-- rows set file fields (`FileHandlingPlugin`);
-- rows create positioned records or move them (`PositioningPlugin`).
+- rows set file fields (`FileHandlingPlugin`), directly or with raw values;
+- rows create positioned records or move them (`PositioningPlugin`), directly
+  or with raw values.
 
 ## HTTP routes
 

package/docs/GUIDE/30-backend-capabilities.md

@@ -37,6 +37,19 @@ See [contributing](../contributing.md#real-databases-and-redis) for native
 database verification commands. A passing SQLite suite is not native PostgreSQL/MySQL
 evidence, even when launched with a different storage-mode environment variable.
 
+## Column values
+
+Only `object` and `array` fields store a structure, as JSON. Every other stored
+field takes a single value: a string, number, boolean, bigint, `Date`, `Buffer`
+or `Uint8Array`, or null. A write, filter or cursor that gives such a field an
+object or array, which a `none` or `blob` field accepts from JSON, fails with
+`RestApiValidationError` (rule `single_value`) naming the field, before any SQL
+runs. MySQL drivers that format queries on the client (mysql2 before 3.17, and
+mysql) would otherwise expand a plain object into `column = value` pairs, so a
+value for one column could set others. Declare structured fields as `object` or
+`array`, or convert the value with a synchronous `storage.serialize` function,
+whose result is stored as it is.
+
 ## Temporal values
 
 Public date/dateTime inputs are strings; time inputs remain strings too. Built-in
@@ -88,8 +101,9 @@ storage slots.
 | --- | --- | --- | --- |
 | Ids of inserted rows | `RETURNING` | `RETURNING` on MariaDB 10.5+; on MySQL the first `insertId` and `auto_increment_increment`, read back and verified (`RestApiBulkInsertIdsError` on a mismatch) | `RETURNING` |
 | Upsert | `ON CONFLICT` | `ON DUPLICATE KEY UPDATE`; a conflict on another unique index fails the batch | `ON CONFLICT` |
-| `lockByIds` | `FOR UPDATE` | `FOR UPDATE` | scoped read (one writer) |
-| `claim` | `FOR UPDATE SKIP LOCKED` | candidates first, then `SKIP LOCKED` windows by primary key (MySQL 8.0.1+, MariaDB 10.6+) | scoped read (one writer) |
+| `lockByIds` | `FOR UPDATE` | `FOR UPDATE` | write lock, then a scoped read |
+| `lockWhere` | `FOR UPDATE`, read again until no new row appears | `FOR UPDATE` | write lock, then a scoped read |
+| `claim` | `FOR UPDATE SKIP LOCKED` | candidates first, then `SKIP LOCKED` windows by primary key (MySQL 8.0.1+, MariaDB 10.6+) | write lock, then a scoped read |
 | Date buckets | named zones and offsets; dateTime columns must be `timestamptz`, as `createKnexTable` creates them | offsets; named zones need loaded time zone tables | offsets only; dates stored as epoch milliseconds or ISO text |
 
 Rows per statement are capped by `chunkSize` (default 500) and by the

package/docs/GUIDE/33-migrating-to-v2.md

@@ -2031,8 +2031,8 @@ empty cursor rejects before SQL. Follow generated links to retain filters,
 fieldsets and the effective size.
 
 `enablePaginationCounts` controls totals for offset pages. With counts disabled,
-offset metadata contains `page` and `pageSize`; it does not promise totals or
-`hasMore`. Cursor pages use one extra row for `hasMore` and never issue a count
+offset metadata contains `page` and `pageSize` (and, since 3.0, `hasMore`, from
+one extra row); it does not promise totals. Cursor pages use one extra row for `hasMore` and never issue a count
 query, regardless of this setting. Ordinary storage previously issued a count
 on the first cursor page and discarded its result; that extra query and its
 count-filter hook invocation have been removed. Collection predicates still