fastify-auto-sqlapi 0.2.4 → 0.4.0

package/docs/adr/0021-upsert-targets-a-verified-unique-key.md

@@ -0,0 +1,153 @@
+# 0021. An upsert targets a unique key verified at startup — MySQL and MariaDB upsert by key, not by `ON DUPLICATE KEY`
+
+- **Status**: accepted
+- **Date**: 2026-09-27
+
+## Context
+
+`upsertMap` turns an insert — main row, secondaries, every bulk item — into an upsert on the
+columns it names. Nothing checked those columns. `buildUpsertRule` accepts any field list, and
+the README's headline example relies on that freedom: a client that knows no ids syncs
+customers by `email` and their tags by `(customerId, tag)`.
+
+Investigating a bug report on bulk upserts showed that the two dialects do not agree on what
+such a rule means, and that MySQL can write to the wrong row without any error:
+
+| Declared key | PostgreSQL (`ON CONFLICT (key)`) | MySQL (`ON DUPLICATE KEY UPDATE`) |
+|---|---|---|
+| has no unique index | every write fails: *no unique or exclusion constraint matching the ON CONFLICT specification* | never conflicts: every upsert inserts, duplicates pile up in silence |
+| is unique, and the row carries another unique key matching a **different** row | unique violation on the other key | updates whichever row conflicts first — possibly the other one, in silence |
+| is unique, row updated through it, key is not the PK | `RETURNING` names the row | `insertId` names no updated row; the PK returned was `insertId + n`, a wrong or non-existent id, and secondaries were attached to it |
+
+MySQL's `ON DUPLICATE KEY UPDATE` has no conflict target at all: it fires on **any** unique
+index of the table, the primary key included. The columns declared in `upsertMap` do not
+select what MySQL matches on. A PK lookup by the declared key (the first fix attempted) is
+itself ambiguous when the key is not actually unique.
+
+The two failure modes that matter are the silent ones — duplicates and wrong-row writes on
+MySQL. PostgreSQL already fails loudly in every case above, just late (at the first write) and
+opaquely (a raw 500).
+
+## Decision
+
+**1. The Schema records the table's unique keys.** The CLI introspects every unique
+constraint and unique index — `pg_index.indisunique` on PostgreSQL,
+`information_schema.STATISTICS` with `NON_UNIQUE = 0` on MySQL — and emits them in the
+generated Schema as `uniqueKeys: [['email'], ['customerId', 'tag']]` (camelCase fields, the
+primary key excluded: it is `primaryKey`). The property is emitted even when empty, so its
+absence means "not generated by a CLI that knows about unique keys", never "no unique keys".
+Left out:
+
+- **partial and expression indexes** (`WHERE deleted_at IS NULL`, `lower(email)`): a column
+  list cannot express them, and PostgreSQL does not accept one as the target of
+  `ON CONFLICT (cols)`;
+- **keys with a column removed by `excludeColumns`**: the API can never write that column, so
+  it can never name the key.
+
+**2. `defineTable` refuses an `upsertMap` rule that is not a verified unique key.** For every
+rule — the table's own schema and every write-join schema — the declared field set must equal
+the Schema's `primaryKey` or one of its `uniqueKeys`, as a set: same fields, any order, nothing
+extra (the rule PostgreSQL itself applies to a conflict target). It is **fail-closed**: a rule
+that cannot be verified — a hand-written Schema, or one generated before `uniqueKeys` existed,
+whose rule is not its `primaryKey` — is refused with a message naming the table and telling to
+regenerate. The check reads the **Schema**, not a table config: a child table written only
+through a write join — `content_info(content_id, lang_id, …)` under `content` — needs no
+`defineTable` of its own; its rule sits in the parent's `upsertMap` and is verified against
+the child Schema's `primaryKey` / `uniqueKeys`. A view has no keys, so a view never takes an
+upsert rule. The check lives in
+`defineTable` with every other config check; a `DbTables` entry written as a plain object does
+not go through it, as it already skips the others.
+
+**3. Every upserted record must carry its key — `400` otherwise.**
+
+- *The key is a natural key* (not the primary key): every key field must be present and not
+  `null`. A `NULL` never conflicts, so such a record could only insert, and could not be found
+  again by its key.
+- *The key is the primary key*: a record **without** any PK field is a plain insert with a
+  generated key — the normal way to mix new and existing rows in one bulk. A PK present only
+  in part, or `null`, is a `400`.
+- **Two items of one request with the same key** are a `400`: PostgreSQL refuses to affect a
+  row twice in one statement, MySQL would let the last one win. Neither is a meaningful
+  request.
+
+The check runs on the record as it reaches the database — after hooks, tenant injection and
+the FK auto-fill of secondaries.
+
+**4. Execution: the same contract, one implementation per dialect.**
+
+- **PostgreSQL** keeps `INSERT … ON CONFLICT (key) DO UPDATE`: once the key is a verified
+  unique key, it targets exactly that key, and a collision on any other unique index is an
+  error — which is the contract. One statement per chunk, as before.
+- **MySQL and MariaDB** stop using `ON DUPLICATE KEY UPDATE` — it has no conflict target on
+  either — and upsert by key explicitly, per chunk:
+  1. **probe**: which items already exist, one `SELECT` for the chunk (a `UNION ALL` part per
+     item, tagged with its position, so value comparison stays the database's). More than one
+     row for one key means the key is not unique in the database (the index was dropped after
+     generation): the chunk is refused before anything is written. The tenant conflict guard
+     stays its own query, as on PostgreSQL: it lives in the engine, which owns the tenant;
+  2. **insert** the new items in one multi-row `INSERT`;
+  3. **update** the existing items by their key in one statement — each column a `CASE` over
+     the keys, the `WHERE` naming the same keys — so it can only reach the rows those keys
+     name. The number of rows matched must equal the number of items; a row deleted since the
+     probe is an error, not a silently skipped write. This relies on the driver reporting
+     matched rows, which `mysql2` does by default (`CLIENT_FOUND_ROWS`).
+
+  A collision on another unique index is then a duplicate-key error there too. PKs come from
+  the probe for existing rows and from the insert for new ones, so no PK lookup after the
+  write is needed.
+
+The observable behaviour is the same on both dialects, and one set of integration tests, run
+against both, is what pins it.
+
+## Alternatives considered
+
+- **Document the pitfalls, change nothing** — rejected: the MySQL failure modes are silent
+  data corruption, which no amount of documentation makes acceptable for a library that
+  advertises upsert-by-natural-key as a feature.
+- **Error only when the PK lookup finds several rows** — rejected as insufficient: it catches
+  a non-unique key after the write, and does nothing for `ON DUPLICATE KEY` matching a row
+  through another unique key.
+- **Reject any record carrying the columns of another unique key** — rejected: it forbids the
+  most common upsert of all, by `id` on a table with a unique `email`.
+- **Probe every other unique key and answer `409` when one matches a different row** — works on
+  both dialects, but adds a query per key per chunk on PostgreSQL, which needs none: its
+  `ON CONFLICT (key)` already errors on those collisions.
+- **Change the signature to `upsert(key, fields)`** — clearer intent, and it would make the
+  key checks of point 3 schema validation. It does not make anything safer on its own: MySQL
+  still matches on any unique index while `ON DUPLICATE KEY` is used, the key must still be a
+  verified unique key, and new rows with a generated PK still have no key to pass. A breaking
+  change for every client, for clarity only — deferred; it can come in a major release on top
+  of this decision without changing it.
+- **Explicit upsert on PostgreSQL too** — one implementation instead of two, but one or two
+  extra queries per chunk on the dialect whose native statement already has the right
+  semantics.
+- **Check the rules at plugin registration** — would also cover plain-object `DbTables`
+  entries, but splits config validation across two places; every other rule is enforced by
+  `defineTable`.
+
+## Consequences
+
+- **Breaking.** An `upsertMap` rule on columns without a unique index now fails at startup,
+  and so does any rule on a Schema that predates `uniqueKeys` unless it names the primary key:
+  regenerate the schemas. Records missing their key, and duplicate keys in one request, become
+  `400`s. Migration notes go in `BREAKING_CHANGES.md`.
+- **MySQL and MariaDB cost more.** An upsert is two to three statements per chunk (probe,
+  insert, update) instead of one, plus the tenant guard where there is one, as before;
+  PostgreSQL is unchanged. The count does not grow with the number of
+  rows in the chunk.
+- **A collision on another unique index is a database error** on both dialects — a sanitized
+  500 ([ADR 0013](./0013-sanitized-db-errors.md)). A descriptive `409` can be added later
+  without changing this decision.
+- **There is a window between probe and write on MySQL and MariaDB.** A concurrent insert of the same key
+  fails with a duplicate-key error; a concurrent delete fails the matched-row check. Both are
+  errors, never a wrong write — consistent with non-transactional bulk
+  ([ADR 0003](./0003-bulk-not-transactional.md)).
+- **A nullable unique column cannot be a natural upsert key for rows where it is `NULL`**: those
+  rows are inserted through a plain insert, or the column made `NOT NULL`.
+- The generated Schema is the source of truth for keys, as it is for columns
+  ([ADR 0017](./0017-the-schema-is-the-contract.md)): an index added or dropped in the database
+  reaches the API when the schemas are regenerated. A dropped index is still caught at runtime
+  by the probe on MySQL/MariaDB and by `ON CONFLICT` itself on PostgreSQL.
+- `QueryClient.bulkInsertOrUpdate` / `insertOrUpdate` get the upsert by key too, so a
+  custom route calling them directly is protected by the probe — but not by the startup check
+  or the `400`s, which need a table config.

package/docs/adr/README.md

@@ -34,6 +34,8 @@ not by editing history.
 | [0017](./0017-the-schema-is-the-contract.md) | The database schema is the API contract — by default, and only by default |
 | [0018](./0018-one-seam-to-the-http-framework.md) | The HTTP framework is reachable through one seam — `ApiRequest` |
 | [0019](./0019-column-exclusion-is-generation-time-config.md) | Removing a column from the API is generation-time config — `excludeColumns`, never a Schema edit |
+| [0020](./0020-no-conditional-or-mass-update-route.md) | No conditional or mass update route — a predicate write is a custom route |
+| [0021](./0021-upsert-targets-a-verified-unique-key.md) | An upsert targets a unique key verified at startup — MySQL and MariaDB upsert by key, not by `ON DUPLICATE KEY` |
 
 ## When to write one
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "fastify-auto-sqlapi",
-  "version": "0.2.4",
+  "version": "0.4.0",
   "description": "Fastify plugin for automatic CRUD API generation from PostgreSQL, MySQL and MariaDB tables",
   "type": "module",
   "main": "./dist/index.js",
@@ -83,20 +83,20 @@
   "homepage": "https://github.com/LucaRainone/fastify-auto-sqlapi#readme",
   "dependencies": {
     "@sinclair/typebox": "^0.34.14",
-    "fastify-plugin": "^5.1.0",
+    "fastify-plugin": "^6.0.0",
     "node-condition-builder": "^1.2.0"
   },
   "devDependencies": {
     "@eslint/js": "^10.0.1",
     "@fastify/postgres": "^6.0.2",
     "@fastify/swagger": "^9.0.0",
-    "@fastify/swagger-ui": "^5.0.0",
+    "@fastify/swagger-ui": "^6.1.1",
     "@types/node": "^22.13.4",
     "@types/pg": "^8.11.11",
     "dependency-cruiser": "^18.1.0",
     "eslint": "^10.8.0",
     "eslint-plugin-sonarjs": "^4.2.0",
-    "fastify": "5.8.5",
+    "fastify": "5.12.4",
     "fuse.js": "^7.5.0",
     "jscpd": "^5.0.14",
     "knip": "^6.31.0",