@lenne.tech/nest-server 12.1.0 → 12.2.0

package/docs/offline-sync-protocol.md

@@ -0,0 +1,410 @@
+# Offline sync protocol
+
+> **Status: implemented** by the sync module (`src/core/modules/sync/`, 12.2.0): pull, push, conflicts, client ids and
+> resync (DEV-3901), hints (§9) and the tombstone purge (DEV-3902), proven end to end by a PWA and a Node test client
+> and by two server instances with skewed clocks (DEV-3903). Specified in
+> phase 1 of the offline-sync epic (DEV-3819, DEV-3898), the open points decided by the cursor prototype of phase 2
+> (DEV-3899). The implementation plan, the port extensions, the reasoning and the measurements live in the framework
+> repository under `.claude/plans/offline-sync/PLAN.md`; §15 lists every decision.
+
+Clients read and write while disconnected and reconcile with the server when they are back: they pull what changed
+since their cursor, push what they changed locally, and receive tombstones for deletions. The protocol is HTTP with
+JSON; a client needs `fetch` and nothing from the server's code. It serves mobile apps (React Native, Capacitor) with
+SQLite, web apps and PWAs with IndexedDB or SQLite-WASM, and desktop or Node clients — on every database the server
+supports (MongoDB, PostgreSQL with any engine, MySQL / MariaDB) with the same observable behaviour.
+
+## 1. Principles
+
+1. **Stateless server.** The server keeps nothing per device. The cursor is opaque, owned by the client, and every
+   device and every browser tab may hold its own.
+2. **The same security pipeline as every other request.** A pull reads through `CrudService` (rights, `@Restricted`,
+   `securityCheck`, tenant policy before every store call, tenant row check after every read, row-level security on
+   PostgreSQL, interceptors); a push writes through it. There is no lean path around it.
+3. **Operation rights are declared for sync.** The roles a project puts on its own routes (`@Roles` on a controller,
+   `roles` passed to `update()` in a controller) do not reach `/sync`. Each sync model declares who may pull, create,
+   update and delete; an operation without a declaration is open to administrators only.
+4. **Every pushed change stands alone.** Each item is applied atomically and independently; one failing item never
+   rolls back another.
+5. **No silent loss.** When the server cannot deliver a consistent delta — a cursor behind the retention horizon, a
+   changed access scope — it says so (§8) instead of returning partial data.
+6. **Opt-in per model.** A project without a sync configuration gets no new field, index, route or code path.
+
+## 2. Concepts
+
+| Term | Meaning |
+|---|---|
+| Sync model | A model the project declared syncable. Only those are reachable under `/sync`. |
+| Record | One row/document of a sync model as the client sees it, read through the normal output pipeline. |
+| `id` | The record's id, chosen by the client for records it creates (§3). |
+| `version` | An integer the server increments on every change of the record, starting at 1. Never set by a client. |
+| `updatedAt` | The server's time of the last change. Informational; ordering never relies on it. |
+| `deletedAt` | `null` for a live record, the deletion time for a tombstone. |
+| Tombstone | What remains of a deleted record: `id`, `version`, `deletedAt`, `updatedAt`, on a tenant-scoped model `tenantId`, and the user-id fields the model declares as its scope (e.g. `ownerId`). The server also keeps its audit fields (`createdAt`, `createdBy`, `updatedBy` — who deleted it); a pull sends none of them. No content field. |
+| Cursor | Opaque string the server returns with every pull page; the client sends it back to continue. |
+| Retention horizon | How long tombstones are kept. A cursor older than the horizon cannot be continued (§8). |
+| Mutation id | A unique id the client gives every pushed change, so a repeated push is recognised (§6.5). |
+
+## 3. Identifiers
+
+**Clients create the ids of records they create offline**, in the one format the model declares:
+
+| Format | Shape | Used by | Generated by |
+|---|---|---|---|
+| ObjectId-hex | 24 lowercase hex characters (`^[0-9a-f]{24}$`) | every model unless declared otherwise — existing models and projects change nothing | the client, as in MongoDB's own model where the driver creates the id — but with a random tail: 4 timestamp bytes plus 8 bytes from a secure random source, fresh for every id (see below); not a driver's default `new ObjectId()` |
+| uuidv7 | 36 characters, `xxxxxxxx-xxxx-7xxx-[89ab]xxx-xxxxxxxxxxxx`, lowercase | a model declared with it — a choice for **new** models only | a uuidv7 generator (`crypto.randomUUID()` is v4) |
+
+* **One format per model, never mixed** in a collection or table (decided 2026-10-29). An existing model does not
+  switch: that would mix both formats in its data.
+* The server validates every id it receives against the model's format and refuses anything else (`INVALID_ID`, §7).
+  It never rewrites a client id.
+* Records the server creates itself (a create through the normal API) get the model's format. Server-generated
+  ObjectIds are predictable from an observed one, so a client could occupy the next ones; on a sync model the server
+  therefore generates a fresh id and retries when its own id is taken, and a normal create never fails on that.
+* Ids of models that are not syncable keep today's rule: ObjectId-hex only.
+* **A client's next ids must not be predictable.** A uuidv7 carries 74 random bits by construction. A driver's default
+  ObjectId does not: its per-process value and counter make every next id of a client predictable from one it has
+  created, so a member of the same tenant who sees one could create those ids first, and the client's own create
+  would then meet a record that is not its own. That is why a client fills the 8 bytes after an ObjectId's timestamp
+  from a secure random source (`crypto.getRandomValues`) for every id — still a valid ObjectId, as unguessable as a
+  uuidv7. The server cannot tell how an id was generated, so this rule is the client's. Beyond it, every access is
+  checked against the caller's rights, an id taken by a record the caller may not see answers `ID_UNAVAILABLE`, and
+  the push rate limit bounds probing. ObjectId stays the default; uuidv7 is a model's choice for new models.
+* Records created offline may reference each other by these ids before any of them reached the server. Push them in
+  dependency order (§6.3) or accept that a reference may point to a record the server has not received yet — the server
+  checks a reference against the caller's rights only where the model declares that check.
+
+How each database stores the formats: the id column is already a string column wide enough for both on PostgreSQL
+(`text`) and MySQL / MariaDB (`varchar(255)`). On MongoDB an ObjectId-hex id is stored as an ObjectId and a uuidv7 id
+as a BSON UUID (binary subtype 4, MongoDB's recommended UUID representation); the core keeps ids as strings, the store
+converts at its boundary. References to a model use that model's representation. Because a collection holds one
+format, casting, populate and sort never meet two representations side by side.
+
+**A create whose id is already taken** by a record or tombstone the caller may see — inside their tenant scope, with the
+model's pull roles, not hidden by `securityCheck` — is answered `conflict` with that record as the caller may see it (§6.5) — without comparing the stored values with `data`: an answer that depended on the comparison
+would let the caller test guesses for fields hidden from them. Taken by a record or tombstone the caller may **not**
+see, the item is refused with `ID_UNAVAILABLE` — the same answer,
+same timing, for every reason an id cannot be used; it carries no tenant, owner, content or deletion state. That an id
+is occupied stays observable — decided (O2, §15): ids stay unique across tenants, the push rate limit bounds probing,
+and an id generated as above (uuidv7, or ObjectId-hex with its random tail) cannot be guessed in the first place.
+
+## 4. Requests and authentication
+
+* **Base path** `/sync` (configurable). Model names in paths are the names the project registered for sync.
+* **Tenant:** with `X-Tenant-Id` a request covers that one tenant (and requires an active membership in it); without the
+  header it spans every tenant the caller is an active member of.
+* **Authentication:** a Better-Auth session cookie or a bearer token (JWT or session token), exactly as for every other
+  route. API tokens are refused on sync routes unless the project releases a sync scope for them.
+* **Writes with cookie authentication** follow the rules of every writing endpoint — the session cookie is
+  `SameSite=lax`, credentialed CORS is answered for the trusted origins (for any origin under `cors.allowAll`, a
+  development setting) — and push additionally requires
+  `Content-Type: application/json` (which forces a CORS preflight; required of every push, `400` otherwise) and,
+  while the server uses cookies, refuses a request whose `Origin` is not a trusted origin or whose `Sec-Fetch-Site` is
+  `cross-site` (`403`). The trusted origins are those credentialed CORS answers plus the app's and the API's own. A
+  client without fetch metadata (a native app, a server) sends neither header and is not affected; a browser app
+  using a bearer token runs on a trusted origin anyway, or CORS would refuse it first.
+* **Session expired while offline:** push answers `401` for the whole request before it looks at any item. Nothing is
+  applied and nothing is discarded; after signing in again the client sends the same items with the same mutation ids
+  and gets no duplicates (§6.5).
+* **Size limits:** push accepts at most `maxPushItems` items (default 100) and the server's JSON body limit; pull pages
+  are bounded by count and by size (§5.2).
+* **Rate limits:** pull and push have their own per-user limits; a refused request answers `429` with `Retry-After`.
+
+## 5. Pull
+
+### 5.1 Request
+
+```
+GET /sync/{model}?cursor={cursor}&limit={n}
+```
+
+| Parameter | Meaning |
+|---|---|
+| `cursor` | Absent: start from nothing (a full state, in pages). Present: continue after this cursor. |
+| `limit` | Records per page, 1..`maxPullLimit` (default 200, maximum 1000). |
+
+### 5.2 Response
+
+```json
+{
+  "items": [
+    { "id": "6720c0…", "version": 3, "updatedAt": "2026-11-02T09:14:03.120Z", "deletedAt": null, "data": { "title": "…" } },
+    { "id": "6720c1…", "version": 5, "updatedAt": "2026-11-02T09:15:44.002Z", "deletedAt": "2026-11-02T09:15:44.002Z" }
+  ],
+  "cursor": "c3luYzpub3RlOjE…",
+  "hasMore": true
+}
+```
+
+* `items` holds live records (with `data`, the record as the normal output pipeline returns it, minus `id`, `version`,
+  `updatedAt` and `deletedAt`) and tombstones (no `data`).
+* A page ends at `limit` records **or** when its serialized size reaches `maxPullBytes` (default 1 MiB), whichever
+  comes first — a page always holds at least one item when one is pending. `hasMore` says whether to ask again
+  immediately.
+* `cursor` is returned on **every** page and is a valid resume point: a client that crashes after storing page n and its
+  cursor continues with page n+1. Apply a page's items and store its cursor in one local transaction.
+* **Ordering and duplicates.** Pages are ordered by the server's change order, not by `updatedAt`. A record may appear
+  more than once across pages (it changed again while the client was paging) — keep the item with the higher
+  `version`. Depending on the database the server runs on, a pull also repeats **unchanged** records (decided in
+  phase 2, P2-2): on MongoDB a run re-reads what changed within a window W before its cursor (on a replica set the
+  server's transaction lifetime — 60 s by default — plus 30 s, on a standalone 30 s; decided in DEV-3901), on SQL
+  databases it never does. Clients compare `version` and need not know which.
+* Without a cursor the client receives every record and every unexpired tombstone it may see, then continues with the
+  last cursor like any delta. A client whose local storage was cleared (Safari removes the storage of a site not
+  installed after seven days without interaction) simply starts without a cursor.
+
+### 5.3 What a pull guarantees
+
+* Every change committed before the pull started and visible to the caller is delivered in some page of the run that
+  starts at the given cursor — regardless of how many server instances wrote it, whether their clocks differ, whether
+  two changes carry the same timestamp, or whether a transaction committed after a younger one was already pulled.
+  The mechanism (decided in phase 2, P §3.4, measured on every target database): on SQL databases a change sequence
+  the server assigns in commit order, on MongoDB the server's clock with a window that a commit cannot outlast. The
+  cursor is opaque either way; a client never reads or compares it.
+* No record and no tombstone of another tenant, and no record the caller's rights or `securityCheck` hide.
+
+## 6. Push
+
+### 6.1 Request
+
+```
+POST /sync/{model}
+Content-Type: application/json
+
+{ "items": [ … ] }
+```
+
+### 6.2 Item
+
+```json
+{ "mutationId": "0190f8b3-…", "op": "upsert", "id": "6720c0…", "baseVersion": 2, "data": { "title": "…" } }
+{ "mutationId": "0190f8b4-…", "op": "delete", "id": "6720c1…", "baseVersion": 5 }
+```
+
+| Field | Meaning |
+|---|---|
+| `mutationId` | Unique per change, generated by the client (uuidv7). The same change sent again carries the same `mutationId`. |
+| `op` | `upsert` (create or update) or `delete` (creates a tombstone; a hard delete is not reachable through sync). |
+| `id` | The record's id (§3). |
+| `baseVersion` | The `version` the client's change is based on. Absent **only** on the item that creates the record. Every later item carries the version the item before it produces — a create gives 1, each applied change adds 1 (§6.3) — so a record created and changed twice offline is pushed as the create, then `baseVersion: 1`, then `baseVersion: 2`. A client may instead fold all unsent changes of one record into a single item. |
+| `data` | The fields to set, validated against the model's input type like any create/update. Absent for `delete`. |
+| `fields` | Optional, for `merge`: the field names the client changed. Without it, every key in `data` counts as changed. |
+
+### 6.3 Processing
+
+* Items are processed **in the order sent**, each in its own atomic step. A batch is not a transaction: on a database
+  without transactions (MongoDB standalone) it could not be one, so it is not one anywhere.
+* An item referencing a record created earlier in the same batch sees that record.
+* `upsert` without `baseVersion` on an id that does not exist creates the record with `version: 1`.
+* `upsert` with `baseVersion` is an **atomic conditional change**: it applies only when the stored version still equals
+  `baseVersion` (`WHERE id = … AND version = …` or the MongoDB equivalent; never read-then-write), and it increments
+  `version`.
+* `delete` with a matching `baseVersion` turns the record into a tombstone through the normal delete pipeline (the
+  model's delete rights, `securityCheck`, audit apply). Deleting a tombstone again is answered `applied`. A delete through
+  any other route of the server leaves a tombstone too, so clients learn of it the same way.
+* An `upsert` on a tombstone is a conflict, never a silent resurrection.
+
+### 6.4 Conflicts
+
+A conflict is a `baseVersion` that no longer matches. What happens is the model's (or field's) strategy:
+
+| Strategy | Result |
+|---|---|
+| `reject` (default) | Nothing changes; status `conflict` with the server's record. The client decides. |
+| `lww` | The change is applied on top of the current state. "Last" means **last to arrive at the server**, never the device's clock. Status `applied`. |
+| `merge` | Field by field: fields the client changed overwrite, fields only the server changed stay. Fields declared `reject` still conflict. Status `merged` with the resulting record. |
+| `custom` | A project hook receives the server record, the client change and the base version and returns the result or a refusal; it runs inside the same pipeline. |
+
+### 6.5 Repeats and parallel tabs
+
+* A repeated create is recognised by its id; a repeated change by its `mutationId`. The server records the outcome of
+  every mutation id for `mutationRetention` (default 30 days) and answers a repeat with the recorded status and the
+  current record — whether the first attempt's response was lost or the client retried after a login. When two tabs
+  send the same change at once, one is applied and the other is answered `retry` while the first is still in progress;
+  sent again, it receives the recorded outcome. Neither a duplicate nor a false conflict results.
+* After `mutationRetention` the server no longer recognises the mutation id. A repeated create meets its own record
+  and is answered `conflict` with it (§3); the client compares it with its local copy and resolves it like any
+  conflict. A repeated change meets its own effect as a version mismatch, and the strategy decides: `reject` answers
+  `conflict` with the server's record, which already contains the change; `lww` writes the same values again; `merge`
+  answers `merged`. Never a duplicate record.
+* Tabs sharing one IndexedDB should still coordinate (Web Locks API or a `BroadcastChannel` leader) to avoid needless
+  traffic; correctness does not depend on it.
+
+### 6.6 Response
+
+`200` with one result per item, in the order sent:
+
+```json
+{
+  "results": [
+    { "mutationId": "0190f8b3-…", "id": "6720c0…", "status": "applied", "version": 3 },
+    { "mutationId": "0190f8b4-…", "id": "6720c1…", "status": "conflict", "version": 7, "record": { … } },
+    { "mutationId": "0190f8b5-…", "id": "6720c9…", "status": "rejected", "code": "VALIDATION_FAILED", "details": [ … ] }
+  ]
+}
+```
+
+## 7. Item statuses and codes
+
+| Status | Meaning | Client action |
+|---|---|---|
+| `applied` | The change is stored (also for a recognised repeat). `version` is the new version. | Store `version`; mark the change sent. |
+| `merged` | Stored after a field merge; `record` is the result. | Replace the local record. |
+| `conflict` | Not stored; `record` is the server's current record (a tombstone when it was deleted). | Resolve, then push again with `baseVersion` = `record.version`, or drop the change. |
+| `rejected` | Not stored, for the reason in `code`. Retrying the same change will not help. | Show or log; drop or correct the change. |
+| `not_found` | An update or delete for a record that does not exist **or that the caller may not see** — the same answer for both. | Drop the local record. |
+| `retry` | Not stored for a transient reason (a lock timeout, a store outage, the same `mutationId` still in progress). | Send the same item again later. |
+
+| `code` (with `rejected`) | Meaning |
+|---|---|
+| `VALIDATION_FAILED` | `data` failed input validation; `details` lists the fields. |
+| `FORBIDDEN` | The caller may not change this record or these fields. |
+| `INVALID_ID` | The id does not match the model's id format (§3). |
+| `ID_UNAVAILABLE` | The id cannot be used (§3); no further detail by design. |
+| `CUSTOM_REFUSED` | The `custom` conflict hook refused the change. |
+
+## 8. Resync
+
+When the server cannot continue a cursor consistently, pull answers:
+
+```
+410 Gone
+{ "code": "SYNC_RESYNC_REQUIRED", "reason": "horizon" }
+```
+
+| `reason` | Cause |
+|---|---|
+| `horizon` | The cursor is older than the tombstone retention horizon; deletions in between may be gone. |
+| `access` | The caller's access changed since the cursor was issued while the caller may still pull the model: a tenant membership removed (on a pull without `X-Tenant-Id`), a role changed, a record moved to another tenant or visibility scope. Once the caller may not pull the model at all, the pull answers `403` instead (§10). |
+| `invalid` | The cursor is malformed, was issued for another model or user, or for an incompatible sync configuration. |
+
+**Client procedure:** keep the unsent local changes, delete every local record of that model, pull without a cursor
+until `hasMore` is `false`, then push the kept changes (they meet the new state through the normal conflict rules).
+After an `access` resync records the client may no longer see are gone, which is the point. The server forces this
+resync rather than sending a removal per record; a per-record signal may follow later.
+
+## 9. Hints
+
+Hints tell a connected client that pulling is worthwhile. They are optional: sync works with pull alone, triggered at
+start, on the `online` event and on focus. The server never assumes a client syncs in the background (the Background
+Sync API exists only in Chromium).
+
+* **SSE** for every web client: `GET /sync/hints?models=note,task` with `Accept: text/event-stream`, authenticated like
+  any request (`models` optional: all models without it; a name that is not offered never matches). Events:
+  `event: change` with `data: {"model":"note","at":"2026-11-02T09:15:44.002Z"}`; a comment line every 25 seconds keeps
+  proxies from closing the stream, and `retry: 3000` tells the EventSource how soon to reconnect. A signed-in caller
+  always gets the stream (`200`); models the caller may not pull simply never send a hint, so the stream reveals
+  nothing about which models exist. `404` when the server switched the SSE channel off.
+* **GraphQL subscription** for clients that use GraphQL: `subscription { syncHint(models: ["note"]) { model at } }`.
+* A hint carries the model name and a time — no record, no id, no tenant. It is sent only to subscribers who hold the
+  model's pull roles and may read the changed tenant's records of that model; bursts are collapsed (default 250 ms).
+* Both channels are long-lived, so the server checks them again during their lifetime like its GraphQL WebSockets: a
+  channel whose user lost access, signed out or changed roles is closed; the client reconnects and is checked anew. An
+  SSE stream also ends before its first hint after the re-check interval (30 s by default) and is reconnected — a hint
+  is never delivered on rights older than that.
+* On reconnect a client pulls once, because hints sent while it was disconnected are not replayed.
+
+## 10. HTTP errors
+
+| Status | When |
+|---|---|
+| `400` | Malformed body or query, unknown `op`, more than `maxPushItems` items. |
+| `401` | Not authenticated or session expired. Nothing was applied. |
+| `403` | On a pull: the caller may not pull the model (pull roles withdrawn), or `X-Tenant-Id` names a tenant they left or that was deactivated — access loss like `410 access`: delete that model's (and tenant's) local records and cursor; the unsent changes cannot be pushed under this account (§11). On a push: a cookie-authenticated push failing the origin rules. A push item lacking the operation's roles is `rejected` / `FORBIDDEN` (§7). |
+| `404` | The model is not registered for sync. |
+| `410` | Resync required (§8). |
+| `413` | The body exceeds the server's limit. |
+| `429` | Rate limit; `Retry-After` says when to try again. |
+| `503` | A store needed for the request is unavailable; retry later. |
+
+Error bodies use the framework's error-code format, sync codes in `#LTNS_0700`–`#LTNS_0799` (`SYNC_RESYNC_REQUIRED`,
+`SYNC_MODEL_NOT_FOUND`, `SYNC_INVALID_REQUEST`, `SYNC_ORIGIN_REFUSED`, `SYNC_STORE_UNAVAILABLE`; 429 is
+`RATE_LIMIT_EXCEEDED`). The `410` body is always the one §8 shows, whatever exception filter the project registered.
+
+## 11. Client duties
+
+* Sync at start, on `online`, on focus, and after a hint; back off exponentially on `429`, `503` and network errors.
+* Generate ids in the model's format with the random part §3 requires, and mutation ids, locally; persist unsent changes with their mutation ids until a
+  terminal status (`applied`, `merged`, `conflict`, `rejected`, `not_found`) arrives.
+* Give every queued change of one record the `baseVersion` the change before it produces (§6.2), or fold them into one
+  item; push them in that order.
+* Store each pulled page and its cursor atomically; compare `version` before overwriting a local record.
+* **On logout delete every local record, the cursors and the unsent changes**, and do the same before another user's
+  first pull. The device holds personal data outside the server; the sync module's `SECURITY.md` lists what to delete
+  and how to keep the credential out of reach.
+* **On `410 access` and on `403` to a pull, delete the local records of that model** (and tenant) and the cursor — the
+  caller lost access, and the device must not keep what the server would no longer send (§8, §10, the module's
+  `SECURITY.md`).
+* Treat a cleared storage as a fresh start (no cursor).
+
+## 12. What the server guarantees
+
+* Pull and push run through the same `@Restricted`, `securityCheck`, tenant and output pipeline as every other
+  request, with the operation roles the model declares for sync (§1); hints reach only callers with the pull roles.
+* A pull never delivers a record or tombstone of another tenant; a push never changes one. An update or delete of it
+  answers `not_found` like an unused id, a create on its id `ID_UNAVAILABLE` — the same answer for every reason, with
+  nothing about the record; that an id is taken stays observable by design (§3, decided O2, §15).
+* Tombstones carry no content field. On a model whose records only certain users may see, the tombstone keeps the
+  user-id fields the model declares as its scope (e.g. `ownerId`, decided O1, §15), so it reaches exactly the users who
+  could see the record — never the rest of the tenant.
+* Hard deletes are not reachable through sync.
+* Changes are never lost by the cursor (§5.3) and never duplicated by retries (§6.5).
+* Idempotency records and rate-limit counters live in infrastructure stores with an explicit expiry condition that
+  every reader checks itself — a TTL index (MongoDB) or the expiry sweep (SQL) only removes what has expired. The
+  retention horizon never expires. Tombstones older than `sync.retention.tombstonesDays` (90 days) are removed every
+  hour under a lease, after the horizon moved past them.
+
+## 13. Compatibility
+
+Without a sync configuration nothing changes: no field, index, route or code path is added, and the test suite of a
+project stays as it is.
+
+## 14. Requirement trace
+
+Every requirement of DEV-3819 and where this document meets it (P = `.claude/plans/offline-sync/PLAN.md`).
+
+| Requirement (DEV-3819) | Where |
+|---|---|
+| HTTP with JSON, fully documented, client needs only `fetch` | §1, §4–§10 |
+| No per-device server state; opaque client-owned cursor, one per device/tab | §1, §5.2 |
+| Clients create ids in the server format, repeated create idempotent after days, format checked | §3, §6.5 |
+| Id collision with a foreign record does not reveal it | §3, §7 (`ID_UNAVAILABLE`), decided O2 (§15) |
+| Pull pages bounded by count and size; full state without cursor; resume after abort | §5.1, §5.2 |
+| Cursor behind the tombstone horizon → clear "resync" (410) | §8 |
+| Optional, exchangeable hint channel: GraphQL subscription and SSE; pull-only works | §9 |
+| Conflict strategies `reject` / `lww` / `merge` / `custom`; LWW by server arrival order | §6.4 |
+| Cookie and bearer auth; cookie push under the CSRF/origin rules of all writes | §4 |
+| Expired session while offline → 401, nothing discarded, resend without duplicates | §4, §6.5 |
+| Parallel pushes from several tabs: no duplicates, no false conflicts | §6.5 |
+| Cleared browser storage starts cleanly | §5.2, §11 |
+| No background push assumed; sync on start, `online`, focus | §9, §11 |
+| Only the `DataStore` port and the neutral query language; differences via `store.capabilities` | P §3.1, P §3.3 |
+| `version`, `deletedAt`, cursor field and index from model metadata; SQL schema evolution adds them | P §3.2, P §3.6 |
+| Optimistic lock as an atomic conditional change, port operation with contract test | §6.3, P §3.3 (E1) |
+| Tombstones filtered from all normal reads incl. `count`, `distinct`, populate | P §3.3 (E2) |
+| Unique indexes do not fail on tombstones (partial; MySQL via generated column) | P §3.3 (E3), P §3.6 |
+| Cursor without loss across instances, clocks, equal timestamps, late commits; prototype decides | §5.3, P §3.4, P phase 2 |
+| Push item-atomic; batch transaction only with `capabilities.transactions`, not the default | §6.3 |
+| Hints emitted by the store after every successful write, via in-process / Redis pub/sub; no change feed (P2-5, §15) | §9, P §3.9 |
+| Idempotency keys, rate-limit counters, retention horizon in infrastructure stores with explicit expiry; cron sweep under `LockStore` | §12, P §3.7 |
+| Every pull/push through the full pipeline; no `pullLean` | §1, §12 |
+| Tombstones carry `tenantId` and only `id`, `version`, `deletedAt`, `updatedAt` | §2, §12; plus declared user-id scope fields, decided O1 (§15); the server keeps the audit fields too (DEV-3900), a pull sends none |
+| Hard delete not reachable through sync | §6.2, §12 |
+| Access loss → client deletes local copies; at least a forced full resync | §8 (`410 access`), §10 / §11 (`403` on a pull) |
+| Integration doc says what a client deletes on logout | §11, the module's `SECURITY.md` and its checklist |
+| Acceptance: the same contract suite green on MongoDB standalone and replica set, PostgreSQL × three engines (and MySQL, MariaDB) | `tests/sync-contract.e2e-spec.ts` (framework repository) |
+| Acceptance: no lost change with parallel writers on two instances with skewed clocks | `tests/sync-clock-skew.e2e-spec.ts` |
+| Acceptance: PWA test client in a browser, and a client without one, pass the same scenarios | `tests/sync-pwa-client.e2e-spec.ts`, `tests/sync-node-client.e2e-spec.ts` |
+| Without sync configuration nothing changes | §1, §13 |
+| Decisions 2026-10-29: both id formats, full resync on access loss, SSE + GraphQL, test clients, prototype decides the cursor | §3, §8, §9, P §11 |
+
+## 15. Open questions
+
+| # | Question | Decided by | Notes |
+|---|---|---|---|
+| P2-1 | Cursor mechanism: change sequence with a commit horizon, or overlap window with `version` comparison | **decided 2026-10-31 (DEV-3899, measured on all seven targets):** SQL — a commit-ordered sequence, the counter taken last (no horizon needed); MongoDB — the server clock with a window W derived from the transaction lifetime | P §3.4, P §5 |
+| P2-2 | Whether a page may repeat an unchanged record | **decided 2026-10-31 (DEV-3899):** on MongoDB yes, within W before the cursor; on SQL no | Clients compare `version` either way (§5.2) |
+| P2-3 | MongoDB collections holding ObjectId and string `_id` side by side: casting, populate, sort, references | **dropped 2026-10-29 (owner, DEV-3899):** one id format per model, never mixed — the question cannot arise | §3 |
+| P2-4 | Partial unique index on MongoDB with `{ deletedAt: { $type: 'null' } }`, on MySQL / MariaDB through virtual live-key columns (both refuse a plain partial unique index) | **decided 2026-10-31 (DEV-3899, measured):** MongoDB filters live records with `{ deletedAt: null }` (accepted on 5.0, 7.0, 8.2, covers a missing field, used by the matching query); PostgreSQL keeps `NULLS NOT DISTINCT` with `WHERE deleted_at IS NULL`; MySQL / MariaDB use the virtual live-key column. Every target refuses a second live value and a second live `NULL` and lets tombstones repeat a value | P §3.6, P §5 |
+| P2-5 | Whether `capabilities.changeFeed` (change streams on a replica set, `LISTEN/NOTIFY` on PostgreSQL) is worth it for hints, measured against CrudService-emitted events | **decided 2026-10-31 (DEV-3899):** no. A feed would add hints only for writes that bypass the store. Those reach a client with its next pull at the earliest (on SQL the sequence trigger numbers them too, on MongoDB they carry no cursor field and are not pulled at all — sync models are written through the store), and hints are optional by design. The store path is needed on MySQL, MariaDB and standalone MongoDB regardless. One source, the store's write events | §9, P §3.9 |
+| O1 | Tombstones on per-user-visibility models | **decided 2026-10-29 (owner):** a tombstone keeps the user-id fields the model declares as its scope (`syncScope`, e.g. `ownerId`) — only user ids, never content | §2, §12; otherwise a tombstone either reaches the whole tenant (ids and deletion times of other users' records) or nobody (a deletion never reaches the owner's other devices) |
+| O2 | Id collision with a record the caller may not see | **decided 2026-10-29 (owner):** the uniform `ID_UNAVAILABLE` stays, ids stay unique across tenants | §3; ids unique per tenant would change every lookup, reference and route by id in the framework |

package/docs/security-overrides.md

@@ -89,6 +89,20 @@ overrides:
   '@graphql-tools/utils@>=12.0.0 <12.1.0': '12.1.0'
 ```
 
+### Advisories no override can fix yet: `minimatch` and `glob` (2026-10-31)
+
+[GHSA-g3hv-75wg-m249](https://github.com/advisories/GHSA-g3hv-75wg-m249) (minimatch `< 11.0.1`, moderate, DoS
+through globparts amplification) and [GHSA-3g3m-2fjr-6pw4](https://github.com/advisories/GHSA-3g3m-2fjr-6pw4)
+(glob `<= 14.0.1`, moderate) are fixed only in new majors that no consumer can take yet: `@nestjs/cli` 12.0.11
+pins minimatch 10.2.6 exactly, ts-morph 28 (`@ts-morph/common` 0.29), nodemon, `@swc/cli` and glob 13 need the 9.x /
+10.x lines, rimraf 6.1.3 needs glob 13 — all on their newest release. An override across the major is no way out
+(minimatch 11 removed `makeRe()` and `Minimatch#regexp`). nest-server suppresses both in its own repository with
+`auditConfig.unusableFixConsumers`, which `check:overrides` re-tests on every run. What it found for its own tree:
+the only production path is `@lenne.tech/nest-server > ts-morph > @ts-morph/common > minimatch`, used by the
+Permissions module's scanner with globs the developer writes; every other path is dev tooling. Your project's tree
+may have other paths, so check `pnpm why minimatch --prod` and `pnpm why glob --prod` before you suppress the same
+way, and remove the entries once your consumers move to minimatch 11 / glob 14.
+
 ### Only on nest-server 11.x: `ws` and `js-yaml`
 
 `@nestjs/graphql` 13 and `@nestjs/swagger` 11 exact-pin vulnerable versions of these two; their 12.x

package/migration-guides/12.0.x-to-12.1.x.md

@@ -10,7 +10,7 @@ working once it declares them itself — which a starter-based project already d
 **PostgreSQL**, in **MySQL 8.4 / MariaDB 11.8** (§7), or in several of them, model by model. The framework's own
 modules — users, Better-Auth, tenants, API tokens, the AI module, file metadata, locks, migrations, health check, Hub —
 run on MongoDB and PostgreSQL behind one `CrudService` and one security pipeline, and the whole e2e suite runs on both;
-on MySQL and MariaDB everything but Better-Auth runs, proven by the parity matrices. A project that stays on MongoDB changes nothing but
+on MySQL and MariaDB everything runs too, proven by the parity matrices — Better-Auth since 12.1.1. A project that stays on MongoDB changes nothing but
 its dependency list.
 
 ## Overview
@@ -165,8 +165,13 @@ documentation:
 - **No row-level security.** Tenant isolation is the store's (the before-layer and the row check after every read).
   A query through `store.native()`, `store.raw()` or a connection of the project's own reaches every tenant's rows
   unless it filters by `tenantId` itself; the boot warns once per source.
-- **Better-Auth is refused at boot** while the user model (`users`) lives on MySQL / MariaDB. Keep it on PostgreSQL or
-  MongoDB (`database.models: { User: '<source>' }`), or sign in through the legacy endpoints only.
+- **Better-Auth runs there since 12.1.1** (12.1.0 refused it at boot while the user model lived on MySQL / MariaDB).
+  Its Kysely adapter runs on the source's `mysql2` pool, its tables are created by its own migrator as the owner, under
+  the schema lock — in the framework's collation `utf8mb4_bin`: the database's default is switched while Better-Auth
+  creates a table and restored afterwards, because MariaDB refuses a foreign key from Better-Auth's tables to `users.id`
+  across collations. The switch locks every table, so it runs only on a boot that creates one and waits at most five
+  seconds behind an open transaction (the boot then fails and the next one retries).
+  Better-Auth's own column types apply (`timestamp(3)`, `varchar(36)` ids). Nothing to change in the project.
 - **An indexed text field holds at most 255 characters** (a longer value answers 400). Indexing an existing text field
   later is a type change — a reviewed SQL migration. A lookup index over four or more text fields indexes a prefix of
   each; such a unique index is refused.
@@ -234,7 +239,7 @@ graphql-upload 17 accepts `@types/express` 4 and 5.
 | Boot refuses the application role | Superuser, `BYPASSRLS`, owner of a framework table, or `TRUNCATE` right | Use a role that owns nothing; the owner goes into `ownerUrl` |
 | `X.find() reaches MongoDB, but X is stored on postgres` | Code calls the Mongoose model of a model routed to SQL | Use `this.dataStore` / `@InjectStore()` |
 | 403 on a tenant read on PostgreSQL | No tenant scope for a tenant-scoped table (RLS) | Run inside a request with a tenant, or as system work (`RequestContext.runWithBypassTenantGuard()`) |
-| `Better-Auth cannot run on MySQL yet` at boot | The user model is routed to a MySQL / MariaDB source | Route `User` to PostgreSQL or MongoDB, or switch Better-Auth off |
+| `Better-Auth cannot run on MySQL yet` at boot | 12.1.0 with the user model routed to a MySQL / MariaDB source | Update to 12.1.1, or route `User` to PostgreSQL or MongoDB |
 | `The application account … holds ALTER, …` at boot | `database.url` connects as an account with schema rights | Connect as a DML-only account; the owner goes into `ownerUrl` |
 | `Updating the path "…" would create a conflict` on MySQL | Two update operators on one path | One operator per path, as on MongoDB |
 | 400 `a value is longer than its column allows` on MySQL | A text field in an index got more than 255 characters | Shorten the value, or leave the field out of the index |

package/migration-guides/12.1.0-to-12.1.1.md

@@ -0,0 +1,125 @@
+# Migration Guide: 12.1.0 → 12.1.1
+
+## Overview
+
+| Category             | Details                                                                                                                                                                                                                                                                                                                                                                                                  |
+| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **Bugfixes**         | Multi-replica fixes: a BullMQ cron schedule lost in Redis is registered again (§1), a cron tick no longer runs twice when replicas fire on either side of a second boundary and `runParallel: false` holds across replicas (§2), Redis channels are subscribed again after an outage (§3), a withdrawn Better-Auth role stops working on every replica at once (§4). `synchronizedUp()` works (it threw on every call) (§5) |
+| **New Features**     | Better-Auth on MySQL 8.4 and MariaDB 11.8 (§6). The offline-sync protocol is specified (`docs/offline-sync-protocol.md`, not implemented yet)                                                                                                                                                                                                                                                           |
+| **Framework API**    | `CoreRedisService.isReady()`, `subscribeChannel()` / `releaseChannel()`; `SynchronizedUpOptions`; protected hooks on `CoreCronJobs` (`bullJobOptions`, `restoreBullSchedulers()`, `bullSchedulerCheckIntervalMs`, …); `CoreBetterAuthUserMapper` implements `OnApplicationBootstrap` / `OnModuleDestroy`                                                                                                 |
+| **Dependencies**     | `nodemailer` 10.2.1 → 10.3.0. Tested optional peers raised: `bullmq` 6.5.0 → 6.5.1, `@aws-sdk/client-s3`, `@aws-sdk/lib-storage`, `@aws-sdk/s3-request-presigner` 3.1158.0 → 3.1159.0 — `pnpm run update` raises them where a project pins them |
+| **Breaking Changes** | None                                                                                                                                                                                                                                                                                                                                                                                                     |
+| **Migration Effort** | None for most projects. Check §2 for cron jobs with `runParallel: false`, §3 if your code subscribes to Redis channels itself, §4 if you subclass `CoreBetterAuthUserMapper`                                                                                                                                                                                                                            |
+
+## Quick Migration
+
+```bash
+pnpm add @lenne.tech/nest-server@12.1.1
+pnpm run update   # nest-server-starter based projects: aligns the pins the framework raised
+pnpm run build && pnpm test
+```
+
+Vendor-mode projects pick it up with the core update (`/lt-dev:backend:update-nest-server-core`).
+
+## 1. A BullMQ cron schedule lost in Redis comes back
+
+With `redis` configured and `bullmq` installed, cron jobs run as BullMQ job schedulers. The schedule was written once
+at bootstrap, so a Redis restart without persistence, a `FLUSHDB` or an eviction removed it, and no replica ran the
+job again until the next deploy. `CoreCronJobs` now registers the schedules the queue no longer lists — after every
+reconnect of the queue connection and every `bullSchedulerCheckIntervalMs` (protected, 60000 ms; `0` switches only the
+timer off) — and logs a warning naming each one. A schedule removed on purpose stays removed, also while replicas of
+the previous version are still running. A tick that fell due while the schedule was missing is not run.
+
+## 2. Cron jobs across replicas
+
+**The tick lease is keyed on the scheduled instant.** A job on a local timer took its lease under the time its timer
+fired. Two replicas firing a few milliseconds apart on either side of a second boundary computed different keys and
+both ran the tick. The key is now the tick's scheduled instant; a manual run (the Hub's "Run now") gets a key of its
+own.
+
+**`runParallel: false` holds across replicas.** It used to hold within one process only, so a long run on replica A
+did not stop replica B from starting the next tick. A distributed job with `runParallel: false` (or `1`) now holds a
+`<job>:running` lease for the duration of the run (60 s, renewed every 20 s, released when the run ends) — on Redis,
+else the SQL lease table, else `cron-locks` — and a replica that finds it held skips its tick. Like the tick lease it
+fails open: an unreachable lease store runs the tick.
+
+**What to check:** a run that never ends now holds the job on **every** replica while its process lives, not only on
+its own. If a job of yours can hang, give it a timeout.
+
+**BullMQ jobs no longer pile up in Redis.** Scheduler jobs carry `attempts: 1`, `removeOnComplete: { count: 100 }` and
+`removeOnFail: { count: 500 }` (protected `bullJobOptions`); before, every completed and failed tick stayed in Redis.
+The first bootstrap after the update removes the backlog an earlier version kept, oldest first, in batches — a tool
+that reads old job history from Redis (Bull Board and the like) shows the last 100 / 500 from then on.
+
+Without `redis`, startup logs one info line naming the jobs that run on every replica because they do not set
+`distributed` (silent with `log: false`). A failed TTL index on `cron-locks` is retried on the next tick.
+
+## 3. Redis outages
+
+`CoreRedisService.isReady()` says whether the shared connection is up (`enabled` only says Redis is configured). At
+boot the service pings once (3 s) and warns when Redis does not answer, then boots anyway.
+
+Channels on the shared subscriber now go through `subscribeChannel(channel)` / `releaseChannel(channel)`, counted per
+holder, and are subscribed again on every reconnect. ioredis re-subscribes only what once succeeded, so a subscribe
+that failed at boot left that replica deaf for its whole life — the framework's own tenant-cache invalidation was one
+such channel.
+
+**What to check:** code of your own that calls `getSubscriber().subscribe(…)` keeps working, but is not re-subscribed
+after a failed first attempt. Switch to `subscribeChannel()` / `releaseChannel()` for that.
+
+GraphQL subscriptions and TUS upload locks still have no fallback while Redis is down — see the "Central Redis" row
+of `.claude/rules/configurable-features.md`.
+
+## 4. Better-Auth roles across replicas
+
+`CoreBetterAuthUserMapper` caches a user's roles and verification state for 15 s. `invalidateUserCache()` cleared
+that cache on the replica that called it only, so a withdrawn role kept working on the others until the entry
+expired. With `redis` configured the invalidation is now broadcast to every replica, which also re-checks the user's
+open WebSockets there.
+
+**What to check:** a subclass of `CoreBetterAuthUserMapper` that implements `onApplicationBootstrap()` or
+`onModuleDestroy()` must call `super`, or the broadcast is neither subscribed nor released.
+
+## 5. `synchronizedUp()`
+
+`synchronizedUp()` threw on every call until 12.1.0: it called `up` on the loaded migration state, which has none. It
+now runs the `MigrationRunner` — the same runner as `migrate up` — under the state store's lock, takes
+`migrationsDirectory` (default `migrations` in the working directory, as for the CLI) and `pattern`, and refuses a
+`MongoStateStore` without `lockCollectionName` rather than running unsynchronized. A relative `migrationsDirectory`
+given to `MigrationRunner` is resolved against the working directory (it failed with `MODULE_NOT_FOUND` before).
+
+## 6. Better-Auth on MySQL and MariaDB
+
+12.1.0 refused to boot with the user model on a MySQL or MariaDB source while Better-Auth was enabled. It now runs
+there: Better-Auth's Kysely adapter on the source's `mysql2` pool, its migrations applied as the owner under the
+database's `GET_LOCK`. Better-Auth names no collation, and MariaDB refuses a foreign key to `users.id` across
+collations, so while it creates a table the framework switches the database's default collation to `utf8mb4_bin` and
+restores it afterwards — only on such a run, with a 5 s lock wait, because `ALTER DATABASE` locks every table.
+
+Nothing to do for a project on MongoDB or PostgreSQL.
+
+## Smaller fixes
+
+- Upload file names from `multerRandomFileName()` come from `crypto.randomBytes()` instead of `Math.random()`.
+- The Hub's log and trace panels answer an empty page with the cursor unchanged when the shared Redis buffer cannot
+  be read, instead of the replica's local buffer under a cursor that belongs to the shared one.
+
+## Compatibility Notes
+
+| Pattern                                                                  | Status                                                                     |
+| ------------------------------------------------------------------------ | -------------------------------------------------------------------------- |
+| Single replica, no Redis                                                 | Unchanged apart from the info line naming non-distributed cron jobs (§2)   |
+| `CronJobs extends CoreCronJobs` without overrides                        | Unchanged                                                                  |
+| A cron job with `runParallel: false` on several replicas                 | Now exclusive fleet-wide (§2)                                              |
+| Own `getSubscriber().subscribe()` calls                                  | Work; switch to `subscribeChannel()` for re-subscription (§3)              |
+| `CoreBetterAuthUserMapper` subclass with own lifecycle hooks             | Call `super` (§4)                                                          |
+| Better-Auth with users on MySQL / MariaDB                                | New — boots now (§6)                                                       |
+
+## Module Documentation
+
+- [Migrate module](../src/core/modules/migrate/README.md)
+- [Hub module](../src/core/modules/hub/README.md)
+- [Better-Auth module](../src/core/modules/better-auth/README.md)
+- Cron, Redis and data sources: `.claude/rules/configurable-features.md` (rows "Cron Job Deduplication", "Central
+  Redis", "Data Sources")
+- [Offline-sync protocol (specification)](../docs/offline-sync-protocol.md)