@smthrs/plan 0.0.0-stage → 1.0.0-rc.4

package/docs/concepts/authoring-ast.md

@@ -0,0 +1,203 @@
+---
+title: "The authoring AST"
+description: "Node and Planned: how a flow body describes topology as pure data, why map transforms and branch decides, and how a closure survives as a digest."
+sidebar:
+  order: 4
+---
+
+`Node` is the shape a flow body describes, and nothing more. Building a node
+records an inspectable, closure-free, JSON-serializable description and executes
+nothing. `Planned` is the strict placeholder that body sees wherever a step
+result will be.
+
+Together they are the layer above the persisted plan. Neither runs anything.
+
+## Building a node runs nothing
+
+```ts
+import * as Node from "@smthrs/plan/Node"
+
+const greeting = Node.map(Node.succeed({ name: "world" }), (value) => value.name.toUpperCase())
+```
+
+`greeting` is a value holding an AST. The mapper has not run, and `succeed` has
+produced no string. The same split as a build system's action declaration: you
+describe the work, and something else decides when to do it.
+
+The AST has nine variants: `Succeed`, `Fail`, `All`, `Map`, `AndThen`,
+`Branch`, `Catch`, `FlowCall`, and `ActionCall`. `Node.Ast` is the union, and
+every node exposes it as `node.ast`.
+
+## Requirements are phantom
+
+A node carries Effect's requirement channel, `R`, and carries it as a phantom.
+Nothing at plan time reads it: the AST, the graph built from it, its key
+material, and every digest are identical whatever `R` says.
+
+It exists so a value that names an implementation it does not carry, an action
+call, can say so in its type, and so the place that finally has to run that
+implementation can demand it. Building a plan stays requirement-free; only
+executing one is not.
+
+## Map transforms; branch decides
+
+`Node.map` is transformation only. The function is digested, not run: it
+executes later, on the real value. A `map` that decides what happens next is a
+`branch` written wrongly.
+
+`Node.branch` takes both arms and evaluates each once, symbolically, against a
+`Planned` placeholder. The resulting ASTs are what the node stores, so a plan
+shows the exit condition and both continuations before it runs. The predicate is
+digested and evaluated at run time on the real value.
+
+```ts
+import { Action } from "@smthrs/flow"
+import * as Node from "@smthrs/plan/Node"
+import type * as Planned from "@smthrs/plan/Planned"
+import * as Schema from "effect/Schema"
+
+const Draft = Action.make("docs/Draft", { payload: { topic: Schema.String }, success: Schema.String })
+const Publish = Action.make("docs/Publish", {
+  payload: { text: Schema.String, urgent: Schema.Boolean },
+  success: Schema.String
+})
+
+const article = Node.bindPlanned(
+  Draft.call({ topic: "durable plans" }),
+  (text: Planned.Planned<string>) =>
+    Node.branch(Node.succeed({ urgent: true }), {
+      if: (flags) => flags.urgent,
+      then: () => Publish.call({ text, urgent: true }),
+      else: () => Publish.call({ text, urgent: false })
+    })
+)
+```
+
+Both arms contribute their requirements, because both arms are topology the plan
+carries. A run takes one of them, but which one is not known until the predicate
+sees the real value, so an execution has to be able to take either.
+
+`Node.catch` is the same idea for failures: the protected graph and the failure
+arm are both stored, and the arm is built once at plan time against a strict
+planned error placeholder. With no schema the whole typed error channel is
+handled; with one, only the values it accepts, and the remainder survives in the
+resulting error type.
+
+## There is no loop node
+
+A plan is always a DAG, so this module has no loop and never will. Repetition
+lives one level up, in what a flow settles with.
+
+`Node.all`, `Node.race`, `Node.any`, and `Node.quorum` have the same boundary. Its width is fixed at plan time. Fanning out
+over something a step discovered is not `all`: end the round and carry the list
+in the next flow's payload, where it is real data.
+
+## Planned values
+
+A `Planned<T>` may be passed into a payload field, into a branch, or into a map,
+and field access is allowed, because it records a reference path. It may never
+be computed on.
+
+Misuse fails twice. The type is branded, so arithmetic on a planned value is a
+compile error. The proxy's `Symbol.toPrimitive`, `valueOf`, `toString`,
+`toJSON`, application, `in`, and enumeration traps throw a `GraphBuildError`
+rather than let a plan be built around `NaN` or `"[object Object]"`, which
+catches template interpolation, `String(value)`, and `JSON.stringify` of a
+payload holding one.
+
+```ts
+import * as Planned from "@smthrs/plan/Planned"
+
+declare const files: Planned.Planned<{ readonly count: number }>
+
+/** Records the path ["count"] and stays a placeholder. */
+const count: Planned.Planned<number> = files.count
+```
+
+Reading `files.count` is a reference. Interpolating it into a string is
+computation, and the refusal names both the node and the recorded path, so the
+message says which value in which body was misused.
+
+JavaScript exposes no trap for `Boolean(value)` or strict identity, so those
+cannot be refused at run time. They reveal only proxy truthiness or identity and
+never the planned result. Use `Node.branch` for a decision on a real value.
+
+`Planned.TypeId` is interned, so a value that crossed a module boundary is still
+recognized. Interning is a recognition aid rather than a capability:
+`Planned.reference` returns a reference only when the value stored under that
+symbol has the complete `{node, path}` shape.
+
+## Payloads become inert JSON
+
+A payload is stored as its inert JSON mirror. A data-valued callable `toJSON` is
+honored, so a `Date` or a `URL` keys as the value it serializes to rather than
+as an empty object. A function or symbol member is dropped from an object and
+becomes `null` in an array. Shared references and cycles clone as they were
+written.
+
+Accessors and unsupported prototypes without `toJSON` fail as `invalid_payload`.
+A `toJSON` that returns its own receiver fails as `cyclic_payload` rather than
+collapsing to an empty object. The clone and the input therefore key identically
+or refuse together.
+
+## Functions survive as digests
+
+The functions an author writes, a mapper, a continuation, a branch predicate,
+live in `WeakMap`s keyed by the AST node they belong to. The AST keeps only a
+`FunctionIdentity` digest of the function's exact source.
+
+Exact source matters, because whitespace inside a string literal is behavior.
+
+A function whose inert captures were declared with `Node.capture` digests those
+captures and gets deterministic identity. Every other function additionally
+carries process-local, per-function entropy, so two indistinguishable closure
+sources fail closed instead of sharing a cache key.
+
+```ts
+import * as Node from "@smthrs/plan/Node"
+
+const suffixed = Node.capture({ suffix: "!" }, (value: string) => `${value}!`)
+```
+
+`capture` refuses a capture record nested past 256 levels with a path-bearing
+error rather than overflowing the native stack, and refuses accessors, exotic
+prototypes, symbols, and cycles for the same reason it refuses them in a
+payload: an identity that cannot describe the function's behavior is worse than
+no identity.
+
+## Recognizing a node
+
+`Node.isNode` recognizes a node this package built, by registration at
+construction. It also recognizes a rehydrated node, an object sharing the node
+prototype whose own `ast` is a well-formed AST, by that shape, because
+[`@smthrs/flow`](/api/flow) hands an AST that crossed a serialization boundary
+back as a node.
+
+`Node.TypeId` is a public string any object can carry and counts for nothing on
+its own. Every combinator that admits a node reads its `ast` as trusted
+topology, so an object carrying the marker on any other prototype, one
+inheriting it from a node, and one whose `ast` is missing, malformed, or cyclic
+are all refused with the same `GraphBuildError` as any other non-node. A proxy
+is judged by the shape it forwards.
+
+## How a flow call joins the plan
+
+`Node.flowCall` takes a mode. `inline` splices the callee's body into the
+caller's plan, `boundary` makes the call one child execution, and `handoff`
+names the next trampoline round.
+
+The child call is `FlowCall{mode: "boundary"}` rather than a second AST tag
+beside `FlowCall`. The three modes are one authoring construct, a call to a
+named flow with a payload, differing only in how the plan joins it, and every
+consumer (`Graph.build`'s expansion test, key material, the interpreter's
+dispatch) already switches on `mode`. A parallel `ChildCall` tag would duplicate
+the flow tag, payload, and declaration side table in every one of them.
+
+## The engine members
+
+`Node.flowCall`, `actionCall`, `declaration`, `continuation`, `mapper`,
+`predicate`, `catchFilter`, and `functionIdentity` exist for
+[`@smthrs/flow`](/api/flow), which owns flow and action authoring. They are
+supported names, not authoring API: they validate nothing, and several of them
+answer `undefined` for an AST that was rehydrated from JSON, because the side
+tables holding the real functions do not survive serialization.

package/docs/concepts/effects-and-conflicts.md

@@ -0,0 +1,135 @@
+---
+title: "Declared effects and conflicts"
+description: "How declared reads and writes become ordering: the overlap pass, the reader-after-writer pass, the three verdicts, and the cycles both passes refuse."
+sidebar:
+  order: 3
+---
+
+Every node declares what it does to the workspace, as paths. Two plan-time
+passes read those declarations and turn them into edges, so a scheduler never
+has to guess whether two pieces of work can run at once.
+
+```ts
+import * as Plan from "@smthrs/plan/Plan"
+
+const effects: Plan.NodeEffects = {
+  reads: [{ _tag: "Glob", include: ["src/**/*.ts"] }],
+  writes: ["dist/bundle.js"],
+  removes: ["dist/stale.js"],
+  boundaryMode: "hard"
+}
+```
+
+Bare string entries are literal paths at both overlap analysis and execution,
+even when they contain `*` or `**`. Wildcards belong in `Glob.include` and
+`Glob.exclude`.
+
+Measuring a path is run-time work, so a digest here would break the no-I/O law
+that makes a plan reproducible.
+
+## What a node produces
+
+`removes` is optional, and a removal mutates the world exactly as a write does.
+Both passes therefore fold `writes` and `removes` into one produced set. A path
+declared as both is refused as `invalid_effects`, because the declaration
+contradicts itself.
+
+`boundaryMode` is `hard` or `expected`. It travels with the declaration into the
+measured boundary at execution time; the plan passes do not interpret it.
+
+## The overlap pass
+
+Declared write sets make overlap detectable before anything runs. The pass
+compares every pair of producing nodes and skips a pair the graph already
+orders, including one ordered through an ordering edge this same pass just
+inferred. Every other overlapping pair is annotated on both members with the
+resolved verdict.
+
+| Verdict     | Effect                                                                            |
+| ----------- | --------------------------------------------------------------------------------- |
+| `serialize` | The default. The later writer gains an ordering edge.                             |
+| `lane`      | Both writers get lane annotations when either asks for one, and no ordering edge. |
+| `fail`      | `compile` fails with `overlap_forbidden`, for flows that promise disjointness.    |
+
+`fail` dominates `lane`, which dominates `serialize`. A flow that promised
+disjointness must not be quietly serialized, and a pair where either side asked
+for a lane must not be silently ordered.
+
+Each annotation also carries the runtime strategy the pair resolved to, where
+`stop-merge` dominates `delay-rebase`. That is what the scheduler does when the
+predicted overlap actually bites.
+
+A node states its own preferences on its draft, through `conflictStrategy` and
+`runtimeStrategy`. The annotation is a property of the _pair_, not of one
+declaration, which is why both preferences are recorded on the node and resolved
+per pair.
+
+Nodes frozen by an earlier generation are annotated on the new node only,
+because their rows are append-only.
+
+## The reader-after-writer pass
+
+The overlap pass compares write sets against write sets, so a node that _reads_
+a path another node _writes_ is ordered by nothing. Reader and writer could be
+admitted in the same wavefront round; the reader would then measure pre-producer
+bytes and, because the dispatch key honestly folds the digest it measured, cache
+that wrong execution as a legitimate one.
+
+The second pass closes that. A reader whose read set overlaps a producer's
+produced set gains an ordering edge to the producer, unless the graph already
+orders it there or an explicit `Ref`/`Pending` dependency path selects a read
+before that writer.
+
+Like a `serialize` edge, the edge enters `dependsOn` and never key material. The
+reader computes the same result either way, and its content dependence is
+already keyed by the hermetic boundary digests measured at dispatch.
+
+## When an edge would close a cycle
+
+An explicit `Ref`/`Pending` dependency path that orders the writer after the
+reader is accepted, including a transitive path. It selects earlier bytes: the
+initial source or an earlier producer's output. The pass preserves that order
+instead of adding a reverse edge.
+
+If only inferred producer or `serialize` edges order the writer after the
+reader, there is no explicit selection of earlier bytes. Adding the required
+reader-after-writer edge closes a cycle, so the plan is refused with `cycle`.
+The message names the reader, the overlapping paths, the producer, and the
+contradictory dependency chain. For example, if the chain contains inferred
+edges and no declared-only path selects the earlier read:
+
+```text
+Plan cycle: node lint reads dist/bundle.js, which node bundle produces, so lint must
+follow bundle, but bundle already depends on lint through bundle -> report -> lint
+```
+
+Reachability through declared `Ref`/`Pending` edges establishes explicit
+sequencing. Declaration order alone does not select earlier bytes.
+
+## Overlap is conservative
+
+`FileSet.overlaps` answers `true` whenever two declarations _might_ share a
+path. Over-serializing costs latency; under-serializing costs correctness, so
+the comparison is deliberately asymmetric: `false` proves that no path can
+belong to both declarations.
+
+Two globs always overlap, and so do a glob and a tree artifact, because
+comparing pattern languages exactly is not worth the risk of being wrong.
+
+Exact paths compare in canonical form: every separator rewritten to `/` and
+Unicode normalized to NFC. Without that, `dist\bundle.js` and `dist/bundle.js`
+would be two spellings of one file that the overlap check could not see, which
+is the same class of hole that `.` segments and empty segments would open.
+[Declare the files a node touches](../guides/declare-file-effects.md) covers the
+vocabulary and the forms `workspaceRelative` refuses.
+
+## What this buys the scheduler
+
+A plan arrives at the scheduler with every predictable conflict already
+resolved into an edge or an annotation. Two nodes with no path between them and
+no annotation between them are safe to run at once, and that fact was
+established before a single byte was read.
+
+The annotations that remain describe the cases a plan cannot resolve alone: two
+lane writers that a runtime has to keep apart, and the runtime strategy each
+pair agreed on if the predicted overlap turns real.

package/docs/concepts/plan-value.md

@@ -0,0 +1,168 @@
+---
+title: "The plan value"
+description: "What a compiled plan is: a keyed graph, two digests, a generation counter, and a snapshot nothing can mutate after the fact."
+sidebar:
+  order: 1
+---
+
+A plan is the whole of what a run intends to do, computed before anything runs
+and expressed as one value. It is a `Plan` struct: an id, the flow it was
+compiled for, a generation counter, two digests, and an ordered list of keyed
+nodes.
+
+```ts
+import * as Plan from "@smthrs/plan/Plan"
+
+declare const plan: Plan.Plan
+plan.planId //    "review-4821"
+plan.flow //      "example/Review"
+plan.generation // 0
+plan.baseDigest // the digest a human approved
+plan.digest //    the digest as of this generation
+plan.nodes //     every keyed node, in material-dependency order
+```
+
+Everything else in this package either produces that value, persists it,
+compares two of them, or supplies the vocabulary its fields are written in.
+
+## The four rules
+
+Four properties shape every design decision here. Read them once and most of the
+API stops being surprising.
+
+### Planning demands nothing
+
+A `Node` carries Effect's requirement channel, `R`, as a phantom. No combinator
+reads it, so building a graph asks for no service at all. The channel states
+which implementations running the plan will need, and the place that runs it is
+where the compiler asks for them.
+
+`Plan.compile` is the one exception, and a narrow one: it asks for Effect's
+`Crypto` service, because it hashes. It asks for nothing else.
+
+### Planning performs no I/O
+
+Nothing here reads a file, a clock, or a network. A node's declared `effects`
+carry read and write _paths_, never digests, because measuring a path is the
+scheduler's run-time job. That is what makes a plan reproducible: compile the
+same drafts twice, on two machines, and you get the same keys and the same
+digest.
+
+### Invalidation is re-keying
+
+A node's key is a function of what it consumes, so an edited declaration
+re-keys that node and its dependent cone and nothing else. There is deliberately
+no reverse-dependency index and no invalidating node visitor in this package,
+and there must never be one: content addressing subsumes both. [Step keys](./step-keys.md) covers what a key folds in.
+
+### A plan grows; it is never rewritten
+
+`Plan.append` adds a pre-keyed subgraph at the next generation. Recorded nodes
+keep their id, key, edges, and generation byte for byte, and the SQL raises
+rather than letting a caller update or delete one. Re-ordering after a
+reconciliation happens by re-keying _future_ steps, never by rewriting history.
+
+## Nodes
+
+Within each generation, `plan.nodes` follows the topological order of material dependencies.
+Inferred reader-after-writer edges can point to later array entries. Schedule
+nodes from the complete `dependsOn` graph.
+
+Each entry of `plan.nodes` is a `PlanNode`:
+
+| Field        | What it holds                                                                  |
+| ------------ | ------------------------------------------------------------------------------ |
+| `id`         | The durable lookup address. Never hashed, so renaming a node re-keys nothing.  |
+| `kind`       | `step`, `agent`, or `merge`.                                                   |
+| `key`        | The computed step key, in the same `key1_` format the engine dispatches under. |
+| `material`   | The declaration the key was derived from.                                      |
+| `effects`    | The reads, writes, removals, and boundary mode this node declares.             |
+| `dependsOn`  | The scheduling edge set, including dependencies on later array entries.        |
+| `conflicts`  | One annotation per overlapping writer no dependency path already orders.       |
+| `strategy`   | This declaration's preferred plan-time verdict for an overlap.                 |
+| `runtime`    | This declaration's preferred response when a predicted overlap actually bites. |
+| `priority`   | The scheduler's tie-break among ready work. Higher runs first.                 |
+| `generation` | Which append added this node. Generation 0 is what `compile` produced.         |
+
+`dependsOn` is wider than the material references alone. It also carries the
+ordering edge a `serialize` verdict added and the reader-after-writer edges that
+put a node behind whoever produces the paths it reads. Ordering edges are
+deliberately not key material: a node serialized behind another still computes
+the same result, so re-keying it would throw away a legitimate cache hit.
+[Declared effects and conflicts](./effects-and-conflicts.md) covers both passes.
+
+`priority` is not key material either, for the same reason. It changes latency
+and nothing else.
+
+## Two digests
+
+`baseDigest` is the digest at generation 0: what a human approved and what a
+running run pins. `digest` advances with every appended elaboration. A control
+plane binds an approval to `baseDigest`, so a plan that grew during the run
+still validates against the decision that admitted it.
+
+Both digests cover node identity, every computed key, the edge set, the conflict
+annotations, the declared effects, the priority, and each node's own conflict
+and runtime strategies. They do not cover anything presentational, and they do
+not cover a diff's attribution report.
+
+The command line prints the same projection: [`smthrs flow plan`](/cli/flow)
+renders the plan card and the approval payload that
+[`smthrs approvals approve`](/cli/approvals) and `smthrs flow execute` accept.
+
+## Generations
+
+`compile` produces generation 0. Every `append` produces the next one, and
+`Plan.generationNodes` reads back only the nodes the newest generation added,
+which is exactly what [`@smthrs/plan-store`](/api/plan-store)'s `PlanStore.append` inserts.
+
+A generation exists because plans elaborate. A step that discovers the shape of
+its own follow-on work cannot state that work up front, so the plan grows to
+hold it. What it must never do is change what an operator already saw, which is
+why growth is the only move available.
+
+## An immutable snapshot
+
+A compiled plan is a deep-frozen snapshot of the drafts it was given. Material
+is stored as the inert JSON mirror its key already covers, so a `Date`, a `URL`,
+or any value with a data-valued callable `toJSON` is stored as the value it
+serializes to. Mutating a caller's draft after compiling cannot change the plan,
+its keys, or its digest.
+
+A material accessor, or a prototype with no JSON representation, is refused as
+`invalid_node` naming the node and the payload path rather than stored by
+reference. A `Planned` placeholder is left intact so canonical serialization
+still refuses it.
+
+## Payloads are plaintext
+
+Material is identity: the body and inputs of a node are hashed into its step
+key and written verbatim as plaintext into `node_json` and the approval card an
+operator reads. This package never redacts them, and redacting after hashing
+would break `Plan.verify`. Keep credentials out of payloads. A secret belongs in
+a layer, a capability, or the environment, resolved at dispatch.
+
+## Where a plan comes from
+
+`Plan.compile` takes `NodeDraft` values. Flow authors do not write those by
+hand: [`@smthrs/flow`](/api/flow) walks a flow body into drafts through
+`Graph.build` and `Graph.drafts`, and hands them here. `Plan.NodeDraft` is the
+type in the middle, and it is the whole contract between the two packages.
+
+[Author a node graph](../guides/author-a-node-graph.md) walks that seam.
+
+## Bounds
+
+One compiled plan holds at most `Plan.maximumPlanNodes` nodes, which is 10,000.
+A plan above the ceiling is refused with `graph_too_large` before effect
+analysis. Analysis also refuses more than 250,000 candidate pairs or 10,000,000
+work units, including overlap comparisons, bitset merges and graph traversal.
+This budget applies across all generations replayed by `verify`. Split plans
+that exceed either budget across flow boundaries.
+
+Reachability is cached in bitsets and updated when ordering edges are inferred.
+Analysis yields periodically so cancellation and other fibers can run. Imported
+plans reuse one effect expansion map and candidate index across generations.
+
+Compilation itself walks with explicit stacks and never recurses per edge, so
+depth is a data structure rather than native stack frames.