joist-codegen 2.3.0-next.8 → 2.3.0-next.81

package/skills/joist-partial-updates/SKILL.md

@@ -0,0 +1,352 @@
+---
+name: joist-partial-updates
+description: Joist ORM partial updates, em.upsert, setPartial, nested collection op semantics, and GraphQL/REST/gRPC save inputs. Use when designing, implementing, reviewing, or debugging Joist partial-update or save-input APIs, especially parent-child graphs, id-less nested rows, required-parent validation errors, or accidental collection removal.
+---
+
+# Joist Partial Update APIs
+
+Use Joist's native partial-update contract from the transport through to `EntityManager.upsert` or `Entity.setPartial`. Do not load an existing graph and write custom reconciliation code when IDs and Joist collection operations can express the update.
+
+Authoritative documentation: <https://joist-orm.io/features/partial-update-apis/>
+
+## Non-Negotiable Rules
+
+1. Preserve entity IDs across read, edit, and write boundaries.
+2. Decide whether every supplied collection is exhaustive or incremental before writing the payload.
+3. Omit a field to leave it unchanged. Do not replace omission with `null`, `[]`, or a generated default.
+4. A collection array is exhaustive by default. Omitted existing children are removed from the relation.
+5. A collection becomes incremental when its child objects use `op`.
+6. If any child has `op`, every child in that collection must have `op`, including newly created children.
+7. An empty array is always exhaustive and clears the collection. It never means "no changes."
+8. Use `{ op: "incremental" }` only as a no-op sentinel when the collection key must be present. Prefer omitting the collection key.
+9. Use `include`, `delete`, and `remove` according to their actual relationship semantics.
+10. Test updates against an already-persisted graph and flush the unit of work. Mapper-only tests cannot catch detached required children.
+11. Authorize every supplied ID and operation. Joist enforces persistence rules, not API permissions or tenant boundaries.
+
+## Partial Field Semantics
+
+Joist distinguishes omission, `undefined`, and `null`:
+
+| Input | Scalar or relation behavior | Collection behavior |
+| --- | --- | --- |
+| Omitted | Leave unchanged | Leave unchanged |
+| `undefined` | Leave unchanged | Leave unchanged |
+| `null` | Unset optional value; required value fails validation | Clear the collection |
+| Value or array | Set/update | Exhaustively replace unless `op` enables incremental behavior |
+
+Do not globally normalize `null` to `undefined`. That destroys the caller's ability to explicitly unset an optional field. Adapt only fields whose transport representation differs from the entity graph.
+
+Transport-specific presence rules:
+
+- GraphQL: preserve the difference between an omitted input field and an explicit `null`. Codegen commonly produces `T | null | undefined`; this is what `DeepPartialOrNull` and the partial APIs are designed to accept.
+- REST/JSON: omission means no change; JSON has no `undefined`, so do not synthesize absent keys while decoding or mapping.
+- gRPC/Protobuf: scalar presence can use `optional`, wrappers, or `oneof`, but ordinary repeated fields do not distinguish omitted from empty. Wrap repeated fields in a present message, use a `oneof`, or use a field mask so "unchanged" cannot decode to `[]`.
+
+## Choose the Collection Contract
+
+### Exhaustive Replacement
+
+Use an exhaustive collection only when the caller intentionally sends the complete desired membership:
+
+```ts
+await em.upsert(Author, {
+  id: "a:1",
+  books: [
+    { id: "b:1", title: "Retained and updated" },
+    { title: "New book" },
+  ],
+});
+```
+
+This removes every omitted existing book from `author.books`. It does not imply hard deletion. If the inverse relation is required, detaching an omitted child can produce validation errors such as `Book author is required`.
+
+Do not use exhaustive replacement when:
+
+- The client does not know every existing child.
+- The UI discarded child IDs.
+- Omitted children should remain untouched.
+- Removed children must be hard-deleted rather than merely detached.
+
+### Incremental Update
+
+Use incremental operations when the request describes additions, updates, removals, or deletions without replacing the whole collection:
+
+```ts
+await em.upsert(Author, {
+  id: "a:1",
+  books: [
+    { id: "b:1", op: "include", title: "Updated title" },
+    { op: "include", title: "New book" },
+    { id: "b:2", op: "delete" },
+  ],
+});
+```
+
+Operation meanings:
+
+| Operation | Behavior |
+| --- | --- |
+| `include` | Create or update the child, and ensure it belongs to the collection |
+| `delete` | Remove the child from the collection and call `EntityManager.delete` |
+| `remove` | Detach the child without deleting it |
+| `incremental` | No-op sentinel that enables incremental semantics for an otherwise empty list |
+
+Use `remove` only when the child can validly exist without this parent or is being reparented in the same unit of work. For a required parent relation, `remove` alone usually creates an invalid entity. Use `delete` when the child should cease to exist.
+
+An actual empty array still clears:
+
+```ts
+await em.upsert(Author, { id: "a:1", books: [] });
+```
+
+For an incremental no-op, prefer:
+
+```ts
+await em.upsert(Author, { id: "a:1" });
+```
+
+If a generic payload builder must emit `books`, use:
+
+```ts
+await em.upsert(Author, {
+  id: "a:1",
+  books: [{ op: "incremental" }],
+});
+```
+
+## Stable Identity End to End
+
+IDs are the reliable identity contract for active nested updates. A nested object without an ID is normally a request to create a new entity.
+
+For an editable parent-child graph:
+
+1. Query IDs for the parent and every editable nested entity.
+2. Include those IDs in API response types.
+3. Store IDs in form or draft state.
+4. Send retained rows as `{ id, op: "include", ...changes }`.
+5. Mark deleted persisted rows in place by switching their `op` to `delete` (or `remove`); keep them in the same array rather than moving their IDs to a separate deletion list.
+6. Send new rows without an ID but with `op: "include"`.
+7. Keep the same discipline recursively for grandchildren.
+
+A robust UI row model carries `op` on the row itself, so a deleted book flips to `op: "delete"`/`"remove"` in place instead of moving to a separate collection:
+
+```ts
+interface EditableBook {
+  id?: string;
+  op?: "include" | "remove" | "delete";
+  title: string;
+}
+
+interface AuthorDraft {
+  books: EditableBook[];
+}
+```
+
+Because each row already mirrors `SaveBookInput` (`id`, `op`, scalars), the draft state can go directly onto the wire as the GraphQL input, with no separate deletion list to reconcile back in. Keep the collection incremental (every row has `op`) once any row does.
+
+Do not key editable rows only by array index. Reordering or deleting an earlier row can assign an existing ID to the wrong logical row. Prefer structured row state with stable IDs. If a textarea represents multiple database rows, either maintain line identity explicitly or document why positional identity is safe for that domain.
+
+Do not infer identity from `sortOrder`, names, unique constraints, or array position. Joist may use configured uniqueness for specific upsert cases such as soft-deleted rows, but do not assume `uniqueBy` will match active id-less children. Send the ID unless the current Joist behavior is intentionally relied on and covered by an integration test.
+
+## GraphQL Shape
+
+Make GraphQL inputs mirror Joist's native graph shape:
+
+```graphql
+enum UpsertOp {
+  include
+  remove
+  delete
+  incremental
+}
+
+input SaveAuthorInput {
+  id: ID
+  firstName: String
+  books: [SaveBookInput!]
+}
+
+input SaveBookInput {
+  id: ID
+  op: UpsertOp
+  title: String
+}
+```
+
+The corresponding output query must expose nested IDs:
+
+```graphql
+query EditAuthor($id: ID!) {
+  author(id: $id) {
+    id
+    firstName
+    books {
+      id
+      title
+    }
+  }
+}
+```
+
+Keep the resolver thin when the schema already matches the entity graph:
+
+```ts
+import { type DeepPartialOrNull } from "joist-orm";
+
+async function saveAuthor(input: SaveAuthorInput, em: EntityManager): Promise<Author> {
+  return em.upsert(Author, input as DeepPartialOrNull<Author>);
+}
+```
+
+The cast is appropriate only after checking that relation names, IDs, scalar fields, and nested collection operations align with Joist's expected graph. A cast does not make an incompatible payload safe.
+
+Thin does not mean unchecked. Before calling `upsert`, authorize the parent and every nested ID for the caller's tenant/scope, and validate that each requested attach, reparent, remove, or delete operation is allowed. This security validation is separate from persistence reconciliation.
+
+If the API also accepts convenience fields, such as a textarea that expands into child rows, use a small boundary mapper:
+
+```ts
+type SaveAuthorConvenienceInput = SaveAuthorInput & { booksText?: string | null };
+
+function toAuthorUpsertInput(input: SaveAuthorConvenienceInput): DeepPartialOrNull<Author> {
+  return {
+    id: input.id,
+    firstName: input.firstName,
+    books: mapBooks(input),
+  } as DeepPartialOrNull<Author>;
+}
+
+function mapBooks(input: SaveAuthorConvenienceInput) {
+  if (input.books !== undefined) return input.books;
+  if (input.booksText === undefined) return undefined;
+  return parseLegacyBooks(input.booksText ?? "");
+}
+```
+
+The mapper should transform representation, not reconcile persistence. It must preserve `books: null` as an explicit clear and preserve omission as `undefined`; do not use `??` when `null` and omission have different meanings.
+
+## REST and gRPC Shapes
+
+Use the same graph contract regardless of transport. For example, a REST PATCH body can be:
+
+```json
+{
+  "id": "a:1",
+  "books": [
+    { "id": "b:1", "op": "include", "title": "Updated title" },
+    { "id": "b:2", "op": "delete" }
+  ]
+}
+```
+
+A gRPC message should model the same operations with an enum and explicit collection presence. A plain `repeated SaveBookInput books` field is unsafe for a partial update because its decoded empty list cannot distinguish "omitted" from "clear." Use a wrapper message inside a `oneof`, or an equivalent field-mask convention:
+
+```proto
+message BookChanges {
+  repeated SaveBookInput values = 1;
+}
+
+message SaveAuthorRequest {
+  optional string id = 1;
+  oneof books_update {
+    BookChanges books = 2;
+    bool clear_books = 3;
+  }
+}
+```
+
+An unset `books_update` means unchanged; a present `books` wrapper carries incremental or exhaustive values; `clear_books: true` explicitly clears. Convert generated enum values to Joist's lowercase strings at the boundary if necessary.
+
+Do not invent transport-specific reconciliation semantics. GraphQL, REST, and gRPC should all produce the same `DeepPartialOrNull<Entity>` graph before calling Joist.
+
+## Implementation Workflow
+
+1. Read the entity metadata and identify required inverse relations and delete behavior.
+2. Trace the complete read-edit-write path, not only the resolver.
+3. Verify every editable output type and query includes stable nested IDs.
+4. Verify client state retains those IDs after parsing, normalization, and form edits.
+5. Choose exhaustive or incremental semantics independently for every nested collection.
+6. Make transport inputs match Joist's graph and expose `op` where incremental updates are needed.
+7. Add only the smallest representation mapper required by the transport.
+8. Call `em.upsert`, `entity.setPartial`, or a thin shared helper that delegates to them.
+9. Flush in an integration test and assert identity, membership, deletion, and untouched fields.
+
+## Failure Signatures
+
+### `<Child> <parent> is required` after updating a parent
+
+Likely cause: an id-less or incomplete child array was interpreted as exhaustive replacement. Joist created new children and detached omitted persisted children, whose required parent relation then failed validation.
+
+Fix: preserve child IDs and either send the complete exhaustive graph or use `op: "include"`/`"delete"` incrementally.
+
+### Duplicate children after each save
+
+Likely cause: existing child IDs are missing from the write payload, so each row is treated as new.
+
+Fix: query, retain, and resend IDs. Do not match by sort order or content on the server.
+
+### All children disappear on a no-change save
+
+Likely cause: a payload builder emitted `children: []`, which is an exhaustive clear.
+
+Fix: omit `children`, or emit `[{ op: "incremental" }]` only if the key is mandatory.
+
+### Incremental update is rejected or behaves exhaustively
+
+Likely cause: only some children have `op`.
+
+Fix: give every child in that collection an operation, including new children.
+
+### GraphQL input does not type-check against Joist
+
+Likely cause: generated nullable input types do not directly match strict entity setters, or the API has convenience fields not present on the entity.
+
+Fix: target `DeepPartialOrNull<Entity>` and use a narrow boundary mapper. Do not weaken unrelated types or write a persistence reconciliation layer.
+
+## Anti-Patterns
+
+Never solve a partial-update mismatch by defaulting to these approaches:
+
+- Loading all existing children in the resolver to assign IDs by array position.
+- Diffing old and new child arrays manually.
+- Appending custom delete markers after comparing database state.
+- Matching active children by `sortOrder`, name, or another mutable field.
+- Assuming a database unique constraint supplies identity to an id-less upsert.
+- Dropping nested IDs from API output, DTOs, parsers, or form state.
+- Mapping an omitted collection to `[]`.
+- Mixing children with and without `op` in one collection.
+- Using `remove` for a child whose parent relation is required.
+- Treating an empty array as an incremental no-op.
+- Converting every GraphQL `null` to `undefined`.
+- Adding backward-compatibility reconciliation without a concrete shipped caller that requires it.
+
+Custom reconciliation is justified only when the API contract fundamentally cannot carry stable identity or Joist operations, and changing that contract is impossible. Document that constraint before adding such code.
+
+## Regression Tests
+
+For a nested update bug, build a persisted graph before invoking the real API boundary:
+
+1. Create a parent with at least three children.
+2. Give one child at least two grandchildren.
+3. Update retained entities using their IDs and `op: "include"`.
+4. Add one new id-less entity with `op: "include"`.
+5. Delete one persisted entity with `op: "delete"`.
+6. Omit another relation and prove it remains unchanged.
+7. Flush and reload.
+8. Assert retained IDs are unchanged, new IDs were created, deleted entities are gone, and no duplicates exist.
+
+Do not re-test `em.upsert`'s own collection semantics (omitted vs `[]` vs `[{ op: "incremental" }]`, `remove` vs `delete`, mixed-`op` rejection) — Joist owns and tests those. Test what is yours: that nested IDs and operations survive the transport boundary and reach the resolver. Do not stop at payload snapshots; the original class of failure often appears only during validation or flush.
+
+## Review Checklist
+
+- Are IDs selected for every editable nested entity?
+- Are IDs retained in client or caller state?
+- Does each supplied collection intentionally use exhaustive or incremental semantics?
+- If incremental, does every child have `op`?
+- Are new children marked `include`?
+- Are deleted persisted IDs sent as `delete`?
+- Is `remove` valid for the inverse relation's nullability?
+- Can any no-change path emit `[]` accidentally?
+- Are omitted fields preserved as omitted or `undefined`?
+- Are explicit `null` values preserved where they mean unset?
+- Is the resolver a thin `upsert`/`setPartial` boundary instead of a reconciliation engine?
+- Does an integration test update a persisted graph and verify stable IDs after flush?

package/skills/joist-reactive-hints/SKILL.md

@@ -0,0 +1,150 @@
+---
+name: joist-reactive-hint
+description: Explains how Joist reactive hints work and what they cost to run — a hint is both a reverse-reactivity trigger (which roots re-run when a hinted field changes) and a forward load hint (what gets populated before the lambda runs), so a rule can pull an entire child collection into memory even when its lambda only reads a parent field. Covers followReverseHint, the two rule shapes (child-rooted reacting up to a parent vs parent-rooted reacting down into a child collection), and how to avoid surprise O(children) loads. Use when reasoning about the performance of a reactive rule/field/property (addRule, hasReactiveField, hasReactiveProperty, hasReactiveReference), when a validation rule seems to load far more rows than expected, when deciding whether to put a field in a reactive hint vs load it on-demand, or when a hot-path mutation (e.g. a bulk job) triggers surprise N×M loads. Keywords: reactive hint, load hint, reverse reactivity, followReverseHint, reverseSubHint, m2o vs o2m reactivity, over-invalidation, populate, "a parent rule that hints a child collection loads the whole collection".
+---
+
+# Joist reactive hints
+
+Reactive hints declare the fields a rule, `ReactiveField`, `ReactiveProperty`, or `ReactiveReference`
+depends on. A hint does **double duty**: Joist walks it in **reverse** to find *which* roots to re-run
+when a hinted field changes, then uses it **forward as a load hint** to populate that data on each root
+before the lambda runs. Most hints are cheap — but either direction can pull an entire child collection
+into memory, and the *forward* load is the surprising one, so both are the focus below.
+
+## TL;DR
+
+A reactive rule/field is declared on a **root entity** with a hint. There are two shapes, and they
+cost very differently.
+
+**Shape #1 — rule on the child, hint reaches *up* to the parent** (e.g. a rule on `Book` with
+`{ author: ["currentDraftBook"] }`). When `Author.currentDraftBook` changes, Joist walks the hint in
+reverse (`followReverseHint`), reversing the `Book.author` m2o into the `Author.books` o2m and
+**loading the whole collection** to enumerate the child roots, then re-runs each book's rule.
+`O(children)` per parent-field change. This is **intuitive**: "change the author, and all of its books
+re-check."
+
+**Shape #2 — rule on the parent, hint reaches *down* into a child collection** (e.g. a rule on `Author`
+with `{ books: ["title"] }`). When a single `Book.title` changes, the reverse walk is **cheap** — one
+`book.author` m2o hop to the one author. But the reactive hint is **also a load hint**: before running
+the author's lambda, Joist populates `author.books`, pulling the **entire** collection into memory —
+*even if the lambda only reads `author.status` and never touches the books*.
+
+> The surprising one is **Shape #2**. `{ books: ["title"] }` reads like "react to book titles," but it
+> *also* means "load all of this author's books every time this rule fires." A lambda whose body is just
+> `return a.status === "active" ? undefined : "…"` still pays to load `author.books`, because the
+> **hint, not the lambda body, decides what gets loaded**.
+
+Neither shows up in tests (behavior is correct; only throughput/memory suffer). Both bite bulk/hot-path
+jobs at scale — Shape #2 especially, because it looks cheap from the reverse-reactivity side.
+
+## Why: reverse reactivity *and* load hints
+
+A hinted-field change triggers two steps, both in `joist-core/build/reactiveHints.js`:
+
+**Step 1 — find the roots (reverse walk).** `followReverseHint` starts at the changed entity and walks
+the hint in reverse, reversing each *traversed relation* (via `reverseSubHint`):
+
+- Reverse of an **m2o** (`Book.author`) is the **o2m** (`Author.books`) — so a **child-rooted** rule
+  reacting to a parent field loads the whole sibling collection just to list the roots (Shape #1).
+- Reverse of an **o2m** (`Author.books`) is the **m2o** (`Book.author`) — so a **parent-rooted** rule
+  reacting to a child field finds exactly one root per changed child (Shape #2's reverse is cheap).
+- A **leaf field** (primitive / enum / m2o read as a value, e.g. `["title"]`, `["currentDraftBook"]`) is
+  only a "react to this field changing" trigger; it is **not** traversed into.
+
+**Step 2 — load the hint (forward populate).** Before running each root's lambda, Joist populates the
+reactive hint *as a load hint* on that root so the lambda can read it synchronously. For a parent-rooted
+rule with `{ books: [...] }`, that is effectively `author.populate("books")` — the **full collection** —
+**regardless of what the lambda actually reads**. This is why Shape #2 is expensive even though its
+reverse walk touched only one author: the cost is in the forward load, not the reverse walk.
+
+So `{ author: ["currentDraftBook"] }` on `Book` compiles to a reactive target
+`{ entity: Author, fields: ["currentDraftBook"], path: ["books"] }` — "when `Author.currentDraftBook`
+changes, load `Author.books` and re-validate each" — while `{ books: ["title"] }` on `Author` loads
+`author.books` on every fire so the rule can run at all. Either way Joist over-invalidates for
+correctness: every sibling is re-checked/loaded even when only one could change the outcome.
+
+## The old/new-value shortcut — and when it does NOT apply
+
+`followReverseHint` has a reference-history optimization (`getInstanceData(c).getReferenceHistory`)
+that pulls in **old + new** values — but only when the **step being walked is itself an m2o/poly**
+(the "`Book.author` moved to a new `Author`" reparent case). It exists so a reparent re-validates
+both the old and new parent.
+
+It does **not** turn a child→`parent.field` reaction into a cheap old/new lookup: there, the walked
+step is the `books` **o2m**, so Joist takes the `relation.load()` (full-collection) branch. The
+`currentDraftBook` field's own m2o-ness is irrelevant — it's reacted to *as a field*, not *traversed*.
+
+Rule of thumb for the cost of one changed field:
+- reverse step is an **o2m/o2o** (you're on the child, parent-field changed) → **loads the full
+  sibling collection**.
+- reverse step is an **m2o/poly** being reparented → loads a couple entities (current + history).
+
+## How to spot / avoid it
+
+- **The hint decides the load, not the lambda body.** A parent rule with `{ books: [...] }` loads
+  *every* book on *every* fire, even if the lambda only reads the parent's own fields. If the rule does
+  not actually need the collection, do not hint it; if it does, know you are paying `O(children)` memory
+  per fire.
+- **For SQL-derived fields, split the hint with `hasAsyncReactiveField`.** It takes two hints — a
+  `loadHint`, whose data is populated into memory and passed to the lambda, and a `reactiveHint`, whose
+  data only *triggers* recalculation but is **not** loaded into memory (the lambda recomputes from SQL
+  instead). Put the large child collections you merely need to react to in the `reactiveHint` so a
+  child-field change still recomputes the value without pulling every sibling into memory.
+- **Root the rule where the frequently-changing field lives.** A rule on `Book` reacting to
+  `author.currentDraftBook` reloads all siblings; the same check rooted on `Author` reacting to its
+  own `["currentDraftBook"]` needs no reverse collection-load (but then detecting anything about the
+  *children* forces you to scan them anyway — so this only helps if the check is about the parent).
+- **Only react to fields that change rarely.** Put a field in the reactive hint only if you truly
+  need to re-run when it changes. If you just need to *read* a value, `await rel.load()` it on
+  demand inside the rule and leave it out of the hint. Reserve the hint for the rare-change trigger
+  that actually needs to fire the rule.
+- **Watch bulk/hot-path mutations.** If a job sets some `parent.fk` once per child (e.g. a bulk
+  copy job setting `author.currentDraftBook` per book), any child-rooted rule reacting to that fk
+  reloads every child's siblings — the same `O(children × siblings)` you'd get from an explicit
+  `parent.children.load()` in the loop.
+
+## Worked examples
+
+### Shape #2 — the surprising one: a parent rule hinting into a child collection
+
+```ts
+// ❌ Rooted on Author. The hint `{ books: ["title"] }` reads like "react to book titles" — but it is
+// ALSO a load hint, so every time ANY book's title changes, Joist loads the author's ENTIRE `books`
+// collection into memory to run this rule, even though the lambda only looks at the author's status.
+authorConfig.addRule(["status", { books: ["title"] }], (a) => {
+  return a.status === "active" ? undefined : "inactive authors need review";
+});
+
+// ✅ The rule only depends on the author's own status, so hint only that — no books are loaded.
+authorConfig.addRule("status", (a) => {
+  return a.status === "active" ? undefined : "inactive authors need review";
+});
+```
+
+If the rule genuinely must react to a child field, keep the hint — but know each fire loads the whole
+collection, so reserve it for collections the lambda actually reads and for child fields that change
+rarely.
+
+### Shape #1 — intuitive but still O(children): a child rule reacting to a parent field
+
+Invariant wanted: "an unpublished book must be one of its author's tracked books — its `favoriteBook` or
+`currentDraftBook`."
+
+```ts
+// ❌ Simple but O(books × siblings) on the bulk copy-job hot path:
+// the job sets author.currentDraftBook per book -> reverse-loads author.books each time.
+bookConfig.addRule(
+  ["publishedAt", { author: ["favoriteBook", "currentDraftBook"] }],
+  (b) => { ... },
+);
+
+// ✅ React only to favoriteBook (which changes rarely — on publish, not per draft edit);
+// read currentDraftBook on-demand so it's not a reactive dependency.
+bookConfig.addRule(["publishedAt", { author: ["favoriteBook"] }], async (b) => {
+  if (b.publishedAt) return;
+  const author = b.author.get;
+  const favorite = author.favoriteBook.get;
+  const draft = await author.currentDraftBook.load(); // read, don't react
+  if (b !== favorite && b !== draft) return `${b} is unpublished but neither the favorite nor current draft`;
+});
+```