@agentproto/workflow-runtime 0.12.0 → 0.13.1

package/README.md

@@ -55,13 +55,16 @@ const { output, bindings } = await runWorkflow({ workflow: wf })
 
 ## Step kinds
 
-Every step reads `Bindings` (`{ input, steps, item?, index? }`) via selectors
-and binds its output under `id`.
+Every step reads `Bindings` (`{ input, steps, item?, index?, run? }`) via
+selectors and binds its output under `id`. `run` (`{ workspace }`) is present
+whenever the host wires `RunWorkflowArgs.workspace` — see **Run workspace
+(AIP-58 §4)** below.
 
 | kind | what it does |
 | --- | --- |
 | `tool` | Resolve a TOOL contract against `candidates` and run it. Cacheable. |
 | `agent` | Spawn/reuse an agent session, send a prompt, wait for the turn. Cacheable. |
+| `artifact` | Hash + copy a file out of `$run.workspace` into `artifactsDir`, recorded as an `ArtifactEntry`. Always cache-aware — see **Run workspace**. |
 | `transform` | Pure in-run compute — combine / filter / shape. No dispatch. |
 | `map` | Run a body once per array element (`bindings.item` / `index`); collect outputs. Fail-fast by default; `onError: "collect"` opts into per-item tolerance. |
 | `pipeline` | Run N items through K sequential stages with **no cross-item barrier**. Same `onError: "collect"` opt-in as `map`. |
@@ -73,6 +76,49 @@ and binds its output under `id`.
 | `group` | Run a list of steps as one unit; output is the last step's. |
 | `subworkflow` | Run a nested workflow with its own isolated bindings. |
 
+### WORKFLOW.md `kind: branch` — exclusive arms + join
+
+`compileWorkflow` compiles a declarative AIP-15 `kind: branch` step (goto-style
+`branches[].next` / `default`, forward-only: every target is a LATER sibling in
+the same step list) into nested runtime `branch` nodes. Arms are **exclusive**:
+
+- sorted by position, the arm targets split the siblings after the branch into
+  arm bodies — an arm's body is its target step up to (not including) the next
+  arm's target; the last arm's body runs up to the **join**;
+- the join is the `join:` sibling if declared, else the step right after the
+  last arm's target (so without `join:` the last arm is exactly one step);
+- exactly one body runs (first truthy `when`, else `default`'s), then
+  execution continues at the join — every step from the join on runs once,
+  whichever arm was taken;
+- no `default` ⇒ a no-match runs the steps between the branch and its first
+  arm target (usually none) and continues at the join;
+- the untaken arms' steps are reported through `onStepSkipped` (the daemon
+  surfaces them as `skipped` in `workflow_status` and as AIP-58 `step.skipped`
+  events).
+
+```yaml
+steps:
+  - id: maybe-render-pdf
+    kind: branch
+    branches:
+      - when: $input.exportPdf
+        next: pdf-render          # arm 1 body: pdf-render, pdf-upload
+    join: publish                 # optional — omit and the join is the step after the last arm
+  - id: pdf-render
+    kind: tool
+    tool: pdf.render
+  - id: pdf-upload
+    kind: tool
+    tool: pdf.upload
+  - id: publish                   # runs once, whether or not the PDF arm ran
+    kind: tool
+    tool: site.publish
+```
+
+`fallthrough: true` restores the legacy (pre-exclusive) semantics — the chosen
+target and EVERY sibling after it run, so an earlier arm also runs every later
+arm. It is incompatible with `join`; prefer the exclusive form.
+
 ## Harness-parity capabilities
 
 The engine reaches parity with a code-first agent harness across five axes.
@@ -224,18 +270,204 @@ await runWorkflow({ workflow: wf, cache, cacheKey: "nightly-review" }) // run 2:
 Both `cache` and `cacheKey` must be set for any caching to happen. Supply your
 own `StepCache` (`{ get, set }`) for an in-memory or custom-backed journal.
 
+Both a declarative `kind: "tool"` step and a `kind: "agent"` step accept
+`cacheable: true` — in WORKFLOW.md frontmatter as well as a TS-authored step.
+
+Inside a `map`, each item caches **independently**: the journal key includes
+the item's `[<index>]` path (nested maps append their own index), so item 1
+changing doesn't invalidate items 0 and 2, and every item of a >1-item map
+gets its own cache entry instead of all items sharing (and stomping) one key.
+
+#### Re-running a failed run from the same cacheKey
+
+Because the journal is written per step as the run progresses — not only at
+the end — a run that throws partway through still leaves every already-
+succeeded cacheable step's output in the journal. Re-invoking `runWorkflow`
+with the SAME `workflow`, `input`, `cache`, and `cacheKey` after a failure
+replays every step whose resolved inputs are unchanged and only re-executes
+the step(s) that failed (or whose resolved inputs changed since the failed
+run):
+
+```ts
+try {
+  await runWorkflow({ workflow: wf, input, cache, cacheKey: "run-42" })
+} catch {
+  // fix the underlying issue, then re-run with the SAME cacheKey —
+  // every cacheable step that already succeeded replays; only the
+  // step that failed (and anything downstream of it) re-executes.
+  await runWorkflow({ workflow: wf, input, cache, cacheKey: "run-42" })
+}
+```
+
+This is a manual retry, not a resumable run object — `runWorkflow` has no
+notion of "the run that failed"; the cacheKey is just a namespace the caller
+re-supplies. A first-class `run.retry` verb that resumes a named run without
+the caller re-threading `workflow`/`input`/`cacheKey` by hand — AIP-58 P5 —
+is a HOST concern, not something this transport-agnostic package implements
+itself: see `@agentproto/runtime`'s `WorkflowRunner.retry()` / the
+`workflow_retry` MCP tool, which owns runId allocation and an always-on
+internal journal (so it works even when the original run never passed a
+`cacheKey` at all — every step it runs is journaled internally either way).
+
+### Run workspace (AIP-58 §4)
+
+A host wires `workspace` (an absolute directory) and `artifactsDir` (a
+sibling directory) onto `RunWorkflowArgs` — one allocation per run,
+`<runsRoot>/<runId>/{scratch,artifacts}` in `@agentproto/runtime`'s
+`WorkflowRunner`. Steps read `workspace` two ways:
+
+- **`$run.workspace`** / **`{{run.workspace}}`** — the same `$…`-ref and
+  mustache grammar `$input`/`$steps.<id>` already use, so it drops straight
+  into a `tool` step's `inputs`, a `gate` step's `args`/`cwd`, or an `agent`
+  step's `prompt`.
+- **`_workflowFsRoot`** — the AIP-16 file-contract convention (`inputs
+  ._workflowFsRoot`, injected by the host); use this form when the same
+  manifest needs to stay AIP-16-conformant independent of this runtime.
+
+Both name the identical path — pick whichever reads better at each call
+site. An `agent` step's own **cwd is unaffected** (AIP-15 F25: it stays the
+owning app's root) — the workspace path only reaches it as *text*, via the
+binding in its prompt.
+
+```yaml
+steps:
+  - id: fetch
+    kind: tool
+    tool: yt.fetch-captions
+    inputs:
+      url: $input.url
+      outDir: $run.workspace        # was: $input.outDir (one fixed, reused dir)
+  - id: clean
+    kind: agent
+    prompt: >-
+      Clean {{item}} and write the result under {{run.workspace}}/cleaned/.
+```
+
+#### `kind: "artifact"` — declaring a run artifact
+
+```ts
+{ kind: "artifact", id: "save", key: "brief", path: "briefs/latest.md", contentType: "text/markdown" }
+```
+
+`path` is read relative to `$run.workspace` (absolute paths, and any path
+that would resolve OUTSIDE the workspace, throw). The step hashes
+(`sha256`) and sizes the file, copies it to `artifactsDir/<basename of path,
+sanitized>` — `path: "briefs/latest.md"` above lands at
+`artifacts/latest.md`, keeping the extension, NOT the bare key
+(`artifacts/brief`) — and binds/reports an `ArtifactEntry` — `{ key, path:
+"artifacts/<basename>", sha256, size, stepId, contentType? }` (`path` here is
+relative to the RUN WORKSPACE ROOT, the parent of `$run.workspace` itself —
+not the source location; always read it back rather than assuming a name).
+If two keys' files share a basename, the second one claimed is prefixed with
+its own sanitized key (`artifacts/<key>-<basename>`) instead of silently
+overwriting the first — deterministic, same result run to run. Pass
+`onArtifact` to `runWorkflow` to observe every one recorded, cache hit or
+fresh.
+
+A declarative WORKFLOW.md manifest may instead declare **`outputsFiles`**
+(AIP-16, amended by AIP-58 §4 with `required`) at the top level — checked
+ONCE, after every top-level step finishes:
+
+```yaml
+outputsFiles:
+  brief:
+    path: "./briefs/<runId>.md"   # <runId> / <workflowId> / <isoDate> interpolate
+    required: true
+```
+
+Present ⇒ copied into `artifactsDir` + reported, same as a `kind: "artifact"`
+step. Absent with `required: true` ⇒ the run throws `MissingArtifactError`
+(`code: "missing-artifact"`, attributed to the last step that ran). Absent
+with `required: false` **or omitted (the default — `required` is opt-in)**
+⇒ a `console.warn`, the run still succeeds.
+
+**This is where a shared/fixed destination stops racing.** The
+Motivation this AIP exists for — two concurrent runs of the same workflow
+overwriting each other's output — is closed by `artifactsDir` being
+per-run: `outputsFiles.<key>.path`'s `<runId>`-interpolated form (or a bare
+fixed path) is *never* written to directly; it becomes the **default
+destination for an explicit `run.publish`** (`@agentproto/runtime`'s
+`WorkflowRunner.publish` / the `workflow_publish` MCP tool), which a host
+MUST refuse unless the run has already `"succeeded"`. See
+`@agentproto/runtime`'s own docs for `publish`/`readArtifact` (the "fetchable"
+half) and the compact `{key, path, size}` projection `workflow_status` shows.
+
+#### Cache/replay interplay — a design decision
+
+A cache hit (`cacheable: true` / journaled) replays a step's **recorded
+output** without re-executing it — but if that output only meant something
+because a FILE existed at some workspace-relative path, and this run's own
+workspace is a **different directory** than the run that originally wrote
+it (AIP-58 §4 "two runs MUST NEVER share a workspace" — replay/re-invocation
+under the same `cacheKey` is always a fresh, disjoint `runId`), a naive
+cache hit would report an output pointing at a file that was never actually
+written into THIS run's workspace.
+
+`kind: "artifact"` (and the `outputsFiles` check, which is really the same
+mechanism unrolled) closes this the way the AIP steers hosts to think about
+runs generally: **copy, never share**. A cache hit for an `artifact` step
+relocates (copies) the file from the ORIGINAL run's `artifactsDir` into the
+CURRENT run's own before returning — the two runs' workspaces stay fully
+disjoint on disk; only the *bytes* are reused, exactly the same way
+`run.replay`'s journal-sourced step reuse is specified to work (AIP-58 §6).
+
+A plain cacheable `tool`/`agent` step gets the same treatment, not a narrower
+one (this used to be a documented gap — see "Cache key vs. the run
+workspace" below for why it had to stop being one):
+
+- **Hashing ignores the workspace's identity.** `hashResolvedInputs`
+  replaces every occurrence of `ctx.workspace` inside the serialized
+  resolved input/prompt (including trailing subpaths, e.g.
+  `<workspace>/cleaned/out.txt`) with a stable placeholder before hashing.
+  Two runs of the same logical step under the same `cacheKey` hash
+  identically even though AIP-58 §4 gives each one a fresh, disjoint
+  workspace directory — without this, ANY step whose input/prompt names
+  `$run.workspace` / `{{run.workspace}}` / `_workflowFsRoot` could never hit
+  the journal at all.
+- **A hit relocates forward.** On a hit, every workspace-relative file/
+  directory the entry recorded (`StepCacheEntry.workspaceFiles` — collected,
+  best-effort, from every string in the step's output that resolved to a
+  real path under the ORIGINAL run's workspace) is copied into the matching
+  path under the CURRENT run's own; the recorded output's path strings are
+  then rewritten from the original workspace onto the current one. A
+  downstream step whose input reads one of those paths (`$steps.<id>.path`)
+  finds the bytes there, not just the (by-then-gone) original run's.
+  Best-effort, same posture as the `artifact` step's own relocation: a
+  source already swept by a host's `scratch/` retention policy is not this
+  run's problem to recover.
+- **Still no cross-step content hashing.** This closes the *workspace-path*
+  gap only — the journal still hashes each step's own resolved inputs, not
+  a transitive hash of everything upstream. A step whose resolved input is
+  an opaque reference that stays textually identical across runs (a fixed
+  relative filename, an id) replays from cache on that basis alone,
+  regardless of whether the value behind that reference changed upstream —
+  same limitation any resolved-input-hash cache has, workspace or not.
+
 ### Step lifecycle callbacks
 
-Pass `onStepStart` and `onStepComplete` to `runWorkflow` to observe progress in
+Pass `onStepStart`, `onStepComplete` and `onStepSkipped` to `runWorkflow` to observe progress in
 real time. The callbacks fire for every step kind; for `agent` steps, start fires
 before spawn and complete fires after the turn (and any output-schema retry loop)
 finishes.
 
+A cacheable step replayed from the journal still fires both callbacks (it
+doesn't vanish from progress), with a third `info` argument of
+`{ cached: true }`; an executed step gets `info === undefined`. The daemon's
+`workflow_status` surfaces this as `cached: true` on the step row, and on the
+AIP-58 `step.started`/`step.succeeded` events' `data`.
+
+`onStepSkipped(stepId, { reason: "branch-not-taken", branchId })` fires, once a
+`branch` decides, for every statically-known step in the arms it did NOT take
+(a `map`/`pipeline`/`subworkflow` step reports its own id; a step id that also
+sits on the taken path is never reported). Inside a `map` item the id is
+indexed (`<id>[<index>]`) like the other two callbacks.
+
 ```ts
 await runWorkflow({
   workflow: wf,
-  onStepStart: (stepId) => console.log("starting", stepId),
-  onStepComplete: (stepId, output) => console.log("done", stepId, output),
+  onStepStart: (stepId, info) => console.log("starting", stepId, info?.cached ? "(cached)" : ""),
+  onStepComplete: (stepId, output, info) => console.log("done", stepId, output, info?.cached ? "(cached)" : ""),
+  onStepSkipped: (stepId, info) => console.log("skipped", stepId, "by", info.branchId),
 })
 ```