@sanity/workflow-engine 0.14.0 → 0.16.0

package/DATAMODEL.md

@@ -0,0 +1,252 @@
+# The engine's persisted data model
+
+The engine owns three standalone document types in the Content Lake:
+
+| Doc type                     | Constructed by                               | Mutated after create                                                                          |
+| ---------------------------- | -------------------------------------------- | --------------------------------------------------------------------------------------------- |
+| `sanity.workflow.definition` | `planDefinitionDeploy` (`src/api/deploy.ts`) | never — deploys are create-only                                                               |
+| `sanity.workflow.instance`   | `buildInstanceBase` (`src/instance.ts`)      | every persist writes the canonical state via `instanceStateFields` (`src/engine/mutation.ts`) |
+| `temp.system.guard`          | `compileGuard` (`src/core/guards.ts`)        | full-body refresh on stage re-entry; predicate lift on retract                                |
+
+These field trees are **our contract with every past and future engine
+version reading the same dataset**. This file is where changes to them are
+declared; the model-surface gates keep undeclared drift red:
+
+- **The shape ledger** — `src/__tests__/model-shapes/model-<N>.json`, one
+  file per `DATA_MODEL_VERSION`, **append-only**: only the current version's
+  file is ever regenerated (`vitest -u` on the gate test), so past model
+  shapes stay checked in verbatim — the historical corpus future upgrade
+  machinery tests against. It pins the field trees of every doc
+  type built through the real constructors, plus the stored grammar's
+  vocabulary (every enum's values, every variant discriminator), so a value
+  rename is as red as a key rename.
+- **Coverage tests** (`src/__tests__/model-surface.test.ts`) walk the stored
+  definition schema and fail when any declared key, discriminator, or
+  instantiable enum value stops appearing in the canonical document — the
+  grammar cannot grow past the fixture silently.
+- **The lifecycle ledger** (`model-shapes/model-<N>.lifecycle.json` in
+  `workflow-engine-test`) drives real instances through the verbs —
+  including the recovery paths (expired-lease sweep, loop-back re-entry
+  adoption, dead-end propagation, abort) — and pins the interior trees a
+  fresh construction can't reach: `StageEntry` / `ActivityEntry` /
+  `SubworkflowEntry` rows, pending effects (claim included), effect
+  history, the idempotency ledger, and ALL `HistoryEntry` variants (the
+  accounting is compile-checked against the union, so a new variant fails
+  the build until the scenarios produce it).
+
+Residual review responsibility: each ledger entry pins one shape per
+variant (the scenario's first), so an _optional_ key that only appears on
+some entries of an already-covered variant — or a conditionally-written
+field no fixture or scenario reaches — still needs a reviewer to demand a
+ledger-visible sample. Fixture width is part of reviewing a model change.
+
+This is **not** definition-content versioning: `version` / `pinnedVersion` /
+`contentHash` track what an author deployed; the model stamp tracks the shape
+of the documents the engine itself writes.
+
+## The stamp pair
+
+The two doc types whose format the engine owns outright — definition and
+instance — carry two numbers (`MODEL_STAMP` in `src/core/model-version.ts`).
+Provenance and compatibility are deliberately separate, because consumers
+sharing a dataset upgrade months apart (Functions lag Studios lag CLIs) and
+"newer" must not mean "incompatible".
+
+**The guard doc carries NO stamp.** `temp.system.guard` is a foreign
+contract: the persisted format mirrors the Content Lake's forthcoming guard
+primitive, and the engine must not grow fields on a shape it doesn't own —
+the lake dictates that format long-term. The guard doc's versioning event is
+the `_type` cutover itself (an engine that predates `system.guard` simply
+never matches the new type).
+
+- **`modelVersion` — provenance ("conforms-to").** The value of
+  `DATA_MODEL_VERSION` at write time: "this document conforms to model N
+  now". Bumps on **every** declared shape change, additive included.
+  Instances are re-stamped on every full persist (and rollback restore).
+  Partial patches deliberately leave the pair alone — they don't normalize
+  the shape, so restamping there would lie.
+- **`minReaderModel` — the reader floor.** The value of
+  `DATA_MODEL_MIN_READER` at write time: the oldest engine model that can
+  safely _interpret_ this document. An **additive** change leaves the floor
+  untouched — older engines keep reading and writing newer docs, so
+  mixed-version fleets interoperate. Only a **breaking** change raises it,
+  and raising it is a declared decision to fence out older readers (fleet
+  must upgrade readers before such writers deploy).
+- **Missing stamp = model 0** (floor 0) — the document was last written
+  before the stamp existed. A `modelVersion` with no `minReaderModel` is
+  malformed foreign data and gates conservatively: the stamp is the floor.
+- **Backward compatibility is unconditional.** An engine at model N reads
+  every stamp `0..N`. There is no supported-version floor for _old_ docs;
+  introducing one is a deliberate future mechanism, never a side effect of
+  a shape change.
+- **Fail hard on `minReaderModel` ahead — and only that.** Every engine read
+  of a stamped doc type (definition, instance) passes through
+  `assertReadableModel` — point reads, list queries, the drained-children
+  projection, snapshot hydration's raw reads. A floor ahead of the engine's
+  `DATA_MODEL_VERSION` throws
+  `ModelVersionAheadError` ("upgrade `@sanity/workflow-engine`") instead of
+  silently misreading a shape this engine predates. Deliberately wholesale:
+  one such document in a read's scope fails the whole read (a listing, a
+  verdict computation) rather than degrading silently. Two structural
+  exceptions: existence-only probes (no shape is interpreted), and lake-side
+  GROQ filters, which evaluate persisted fields server-side before any
+  client gate can run — their safety rests entirely on rule 3's
+  additive-only constraint.
+- **Old writers restamp down — the round-trip rule.** An older engine
+  persisting a newer-but-compatible instance stamps its own `modelVersion`
+  (honest conforms-to; the stamp oscillates in mixed fleets, harmlessly).
+  This is safe because full persists are set-only patches and mutation
+  copies are spreads, so fields the older engine doesn't know **survive the
+  round-trip**. That is a load-bearing invariant: an additive change whose
+  fields would NOT survive an old writer's round-trip is not additive — it
+  is breaking, and must raise the floor.
+- **The floor is raise-only.** A rewrite writes
+  `max(stored minReaderModel, own DATA_MODEL_MIN_READER)` — it never lowers
+  or drops the stored floor, because the preserved unknown fields of a newer
+  model may still require it. (The failure mode this fences off is
+  documented across the industry: a rewriter that drops the compatibility
+  gate it doesn't understand silently reopens the door the floor closed.)
+- **Raising the floor is an availability decision, sequenced readers-first.**
+  Before any writer that raises `DATA_MODEL_MIN_READER` deploys, every
+  reader in the fleet — customer-deployed Functions foremost — must already
+  run an engine at or above the new floor (the BACKWARD deployment order:
+  upgrade consumers before producers). And like every engine-side check the
+  floor is advisory — it gates our own clients; the Content Lake stays the
+  sole enforcement point.
+- **The gate ships from the engine.** Consumer surfaces that read these docs
+  without routing through the engine (the CLI's direct reads, MCP query
+  projections, the reactive studio/sdk adapters) adopt `assertReadableModel`
+  with the parse-don't-cast follow-up; until then they render newer-model
+  docs unchecked.
+
+## Rules for changing the model
+
+1. **Any change to a persisted field tree is a model change** and must be
+   declared: update the model-surface snapshot, add an entry to the model
+   log below, and decide — in the same change — whether `DATA_MODEL_VERSION`
+   moves **and whether the reader floor moves with it**. Nothing lands
+   "incidentally".
+2. **Additive optional fields are allowed** without machinery: bump
+   `DATA_MODEL_VERSION`, leave `DATA_MODEL_MIN_READER` untouched, and
+   declare the tolerant read (what an absent value means on docs written
+   before the field existed) plus why the field survives an old writer's
+   round-trip.
+3. **Renames, removals, and semantic changes are not allowed yet.** They
+   require versioned upgrade machinery (per-model normalization steps applied
+   on read) that does not exist; until it does, the model may only grow
+   additively. When such a change eventually lands, it raises
+   `DATA_MODEL_MIN_READER` — a declared fence against older readers, shipped
+   only after the fleet story is written.
+4. **`DATA_MODEL_VERSION` moves when compatibility reasoning changes** — when
+   a reader would need to know which shape it is looking at. A purely
+   internal refactor that provably writes an identical tree does not bump.
+5. **Classify by misinterpretation, not parseability.** The snapshot gate
+   catches structural drift only. A change can be structurally additive yet
+   change the MEANING of existing fields — old readers parse it fine and
+   silently misinterpret. That judgment (would an old reader misread, not
+   just fail to parse?) is exactly what decides whether
+   `DATA_MODEL_MIN_READER` moves, and no snapshot can make it — each model
+   log entry states it explicitly.
+
+## Model log
+
+### Model 0 — pre-governance
+
+Everything written before the stamp existed. Documents carry no
+`modelVersion`. The shape history of model 0 was absorbed by tolerant reads
+that predate this file — recorded here so they are declared compatibility
+rules, not folklore:
+
+- **Instances without `subworkflows[]`** (persisted before the subworkflow
+  registry existed) — read as `[]`; the next full persist materializes it.
+- **Instances without `processedRequests[]`** (persisted before the
+  idempotency ledger existed) — read as `[]`; the next full persist
+  materializes it.
+- **Definitions without `contentHash`** (deployed before content-addressing)
+  — always read as changed, so the first redeploy after upgrade mints a
+  fresh version.
+
+### Model 1 — stamping introduced; actions-only payload model (reader floor: 0)
+
+The `modelVersion` + `minReaderModel` pair is stamped on the definition and
+instance doc types; the guard doc deliberately carries none (foreign lake
+contract — see "The stamp pair"). The floor is 0: model-0 documents keep
+reading exactly as before, and every engine from the amendment below onward
+reads model-1 documents fine (the amendment's sanction — no readers or
+writers existed outside this repo — is what made the pre-amendment window
+irrelevant).
+
+**Amended in place before any consumer existed** (a sanctioned exception to
+rule 3 — model 1 had no readers or writers outside this repo when the
+reshape landed, so the freeze point moved rather than minting a model 2):
+actions became the only payload mechanism, and the persisted trees reshaped
+accordingly. One caveat was weighed and accepted when the exception was
+confirmed: released engines predating the amendment stamp model 1 while
+writing the pre-amendment shapes, so documents they wrote pass the stamp
+gate and then fail ungoverned (a raw shape error, not
+`ModelVersionAheadError`). Those documents are declared disposable — every
+dataset holding them is a this-repo test dataset and must be reset before a
+post-amendment engine touches it. Nothing migrates them.
+
+Definition tree (the stored authoring model):
+
+- Activities carry no payload: `activation`, `ops`, `effects`,
+  `completeWhen`, `failWhen`, `subworkflows`, and `kind` are gone from the
+  activity tree. `target` survives as the off-system marker.
+- Actions gained `when` (a cascade-fired trigger condition), `spawn` (the
+  subworkflow block formerly on the activity), and `roles` (stored verbatim
+  only on cascade-fired actions — the execute pin; fireAction actions still
+  fold roles into `filter` at desugar).
+- Transitions are pure edges: `{name, when, to}` plus presentation —
+  `filter` renamed to `when`, `ops`/`effects` gone.
+
+Instance tree:
+
+- `effectsContext` renamed to `context` (entry `_type`s
+  `effectsContext.<kind>` → `context.<kind>`); it now holds ONLY the
+  start-time seed and a parent's spawn handoff. Completed effects' outputs
+  live solely on `effectHistory[].outputs` (the `$effects` read derives from
+  there).
+- `ActivityEntry` statuses lost `pending` (entries are `active` from stage
+  entry, or `skipped`) and gained `firedActions: string[]` — the per-visit
+  ledger of cascade-fired actions.
+- `SubworkflowEntry` gained required `action` (the spawning action's name;
+  rows key per activity + action).
+- History: the `activityActivated` variant is gone; `actionFired` gained
+  `triggered?: true`; `opApplied` lost its `transition` origin key;
+  `EffectOrigin.kind` is only ever `'action'`.
+
+**Pre-adoption model-1 amendment: moved `applicableWhen` to `start.filter`,
+added `start.kind` — a deliberate freeze bypass; no external consumers
+exist.** Same sanction as the amendment above (rule 3 untouched, the freeze
+point moves, no model 2 minted). The definition root gained an optional
+`start` block: `kind` (`'interactive' | 'autonomous'`, stored explicitly —
+desugar fills the default; the block itself stays absent on definitions that
+never declared one, and an absent block reads as interactive with no filter)
+and `filter` — the former `applicableWhen`, re-scoped from a pure
+candidate-doc predicate to the start-filter context (`$tag` / `$definition` /
+`$now` / `$fields` bound, `*[...]` reading the workflow dataset; the
+`startFilterVars` vocabulary joins the ledger). `start.filter` is a READ-SIDE
+visibility rule — evaluated by `definitionsForDocument`/applicability and the
+Studio start control to decide what appears in Start lists; `startInstance`
+never reads it and never throws on it. An earlier iteration of this amendment
+added a `startFilterOverridden` instance-history variant (paired with a
+throwing start-time gate and an `overrideStartFilter` verb arg); that gate was
+removed before any consumer existed, so this same in-place amendment ALSO
+removed the `startFilterOverridden` variant — no gate, no override, nothing to
+audit. The instance tree is back to its pre-amendment history variants.
+Would an old reader misread (rule 5)? No reader exists outside this repo —
+which is the sanction; post-amendment engines read pre-amendment model-1
+definitions fine (no `start` block ⇒ the defaults above). The released-engine
+caveat above covers this amendment identically: definitions a released engine
+wrote with `applicableWhen` stamp model 1 and then fail the strict parse
+ungoverned — same disposition, those datasets are this-repo test data and are
+reset, not migrated.
+
+## Pending governed changes
+
+- **`temp.system.guard` → `system.guard`** — the guard doc type's `temp.`
+  prefix drops when the lake's own guard primitive ships (see
+  `GUARD_DOC_TYPE` in `src/types/authorization.ts`). A breaking type change
+  for already-persisted guard docs; it must land as a declared model change
+  with a migration story, through this process.

package/README.md

@@ -38,8 +38,8 @@ different at each — read it by where it sits:
    lightweight `{type, name}` shapes for the value's structure, mirroring
    Sanity's `object.fields` / `array.of`.
 3. **Object value expressions** — a `{type: 'object', fields: {...}}` value
-   expression in an op or `initialValue`: one expression per key of the
-   computed object value.
+   expression in an op payload: one expression per key of the computed
+   object value.
 4. **Resolved runtime values** — `fields` on a workflow instance document: the
    declared entries with their current values, written by ops and edits.