@jr2/orchestrator 0.4.1 → 0.5.1

package/docs/concepts/durable-runs.md

@@ -0,0 +1,189 @@
+---
+title: Durable runs
+description: How a run survives a restart of the Orchestrator, what ends it, and how long jr2 keeps a run that ended.
+---
+
+# Durable runs
+
+A run is durable. The [Orchestrator](../reference/glossary.md#orchestrator) saves each run after each transition of its
+[Machine](../reference/glossary.md#machine). When the Orchestrator starts again, it continues each run that did not end,
+from the state that it saved. A run that ended stays readable for one week.
+
+## Why runs are durable
+
+A run can take a long time. An [Agent](../reference/glossary.md#agent) can work for an hour, and a run can wait at a
+[Gate](../reference/glossary.md#gate) for days, until a person answers. In that time, the Orchestrator can stop:
+
+- Each [`jr2 up`](../reference/cli.md#up) that changes the [Kit version](../reference/glossary.md#kit-version) starts
+  the Orchestrator again on the image of the new version.
+- The node of the Orchestrator can fail.
+- The process can stop because of an error.
+
+If a run ended each time that its process stopped, a long run would not finish. Thus the run is not in the process. The
+process holds the run only while it runs it, and the store holds the run between two processes.
+
+## Snapshots
+
+After each transition, the Orchestrator writes the snapshot of the run to its store. The store is a sqlite file on the
+volume of the [Instance](../reference/glossary.md#instance), in the namespace of the Instance. One row holds the whole
+run. With the snapshot, the row holds the facts that a new process needs to continue the run:
+
+- the [Admission](../reference/glossary.md#admission) of each Agent's [Turn](../reference/glossary.md#turn): where the
+  Turn's stream is, and how much of it the run read
+- each timer that waits, with the time at which it fires
+- each [Effect](../reference/glossary.md#effect) that started, and its result when it has one
+- each pick that waits for its answer
+
+The write is complete before the Machine does its next action. A write survives a crash of the process. A power loss of
+the node can lose the last transitions. The Orchestrator then continues from the snapshot before them, as it does after
+each restart.
+
+Only one Orchestrator writes to the store. The Orchestrator has one replica, and a new pod starts only after the old pod
+stopped.
+
+## Restart and re-attach
+
+When the Orchestrator starts, it reads each run that did not end, one at a time. For each run, it:
+
+1. Finds the [Assembly](../reference/glossary.md#assembly) that the run holds: the Assembly that was current when the
+   run started. Then it finds the [Workflow](../reference/glossary.md#workflow) in that Assembly, by the name of its
+   file in `workflows/`.
+2. Compares the shape of the Machine with the shape that the snapshot records. See
+   [Machine shape and drift](#machine-shape-and-drift).
+3. Starts the Machine again from the snapshot.
+
+Thus a run continues on the Machine that it started on, also when `jr2 up` published a newer Assembly after it started.
+When the Orchestrator has no current Assembly, it continues no run. This occurs after its volume lost the Assembly, and
+after `jr2 up` started a new Kit image and before it published. The Orchestrator keeps the live runs, and the next
+publish continues them.
+
+The parts of the run then continue:
+
+- **Agents.** A Turn does not stop when the Orchestrator stops. It continues in its
+  [Harness](../reference/glossary.md#harness). The new Orchestrator re-attaches to the stream of the Turn, from the
+  point that the snapshot records. It does not send the prompt again. The one exception is a Turn whose Admission the
+  Orchestrator did not save before it stopped. That Turn can start again.
+- **Workspaces.** The Orchestrator watches each [Sandbox](../reference/glossary.md#sandbox) of the Instance. When the
+  Sandbox of a [Workspace](../reference/glossary.md#workspace) is still there, the Workspace continues. When the Sandbox
+  is gone or its pod ended, its [Worktrees](../reference/glossary.md#worktree) and the commits that nobody pushed are
+  gone too. The Workspace then sends [`workspace.lost`](../reference/mechanism-events.md#workspace-lost) to its body,
+  and the Machine decides what to do. jr2 never makes the Sandbox again for that Workspace.
+  [Workspaces](workspaces.md#lease-and-continuity) explains how the operator decides to remove a Sandbox.
+- **Timers.** A timer, such as an [xstate](https://xstate.js.org) `after`, fires after the time that remained. A timer
+  whose time passed while the Orchestrator was stopped fires at once.
+- **Effects.** jr2 runs an Effect a second time only when that is safe. When the Orchestrator cannot find out if an
+  Effect finished, the result of the Effect is `unknown`. [Effects](effects.md) gives the rules.
+
+A run that the Orchestrator does not continue gets one of these results:
+
+| Result    | When                                                                                                        | What stays                                         |
+| --------- | ----------------------------------------------------------------------------------------------------------- | -------------------------------------------------- |
+| `lost`    | No Workflow has the name of the run's file, or the row holds no snapshot.                                   | The row and the reason, for one week.              |
+| `drifted` | The Orchestrator cannot read the Assembly of the run, or the Machine of the Workflow has a different shape. | The row and the snapshot.                          |
+| `failed`  | The restore of the run stopped with an error.                                                               | The run stays live. The next start tries it again. |
+
+The Orchestrator prints the ids of these runs on the line that it writes when it starts. A run that it continued is
+normal, and it does not print it.
+
+## Machine shape and drift
+
+The [Machine shape](../reference/glossary.md#machine-shape) is the part of a Machine that a snapshot must agree with:
+
+- the states, their ids, their nesting, and the initial state of each level
+- the type of each state
+- the invokes of each state, by their `id` and their `src`
+- the shape of each Machine that a state invokes or spawns
+- each transition: its source state, its event and its targets
+
+Each save records a digest of the shape. Each restore calculates the digest of the Machine that it loaded, and compares
+the two. When they are different, the run has drift. The Orchestrator does not continue it.
+
+The digest does not include the bodies of the guards and the actions (other than what an action spawns), the `assign`
+calls, the prompts and the descriptions. A change to these parts does not cause drift. A run that continues after such a
+change uses the new code from its next transition.
+
+A run holds its Assembly, so a change that you publish does not change the Machine of a live run. Drift occurs when the
+Orchestrator cannot read the Assembly that the run holds:
+
+- The Assembly is for a different Kit version. After you upgrade the Kit version, the first publish marks each run of an
+  older Assembly `drifted`.
+- The volume of the Orchestrator lost the Assembly.
+- The Assembly does not load.
+
+The Orchestrator keeps a drifted run, and you can read it. [`jr2 status`](../reference/cli.md#status) shows its status,
+`drifted`, with the reason. The reason names the Workflow and the two digests, or the Assembly and why it cannot load.
+[`jr2 runs`](../reference/cli.md#runs) does not show it, because `jr2 runs` shows only the runs that run now. jr2 cannot
+move a drifted run to a new Machine. Thus, before you change the Kit version, let the runs end, or cancel them.
+
+## CANCEL and stop
+
+A run ends when its Machine gets to a final state, when it has a fault, or when you cancel it. A restart of the
+Orchestrator does not end a run. It only stops the process that held the run, and the run stays live in the store.
+
+To end a run yourself, send it the `CANCEL` event with [`jr2 send`](../reference/cli.md#send). `CANCEL` does three
+things:
+
+1. It stops the Machine. The Turn of each Agent of the run ends too.
+2. It saves the run as `cancelled`. The Orchestrator does not continue a cancelled run after a restart.
+3. It deletes the Sandbox of each Workspace of the run, because nothing can use it now.
+
+With `--keep`, `CANCEL` does not delete the Sandboxes, so that you can look inside them. Nothing renews their
+[Lease](../reference/glossary.md#lease), so the operator removes each one after its idle timeout of 30 minutes.
+
+A run that has a fault ends with the status `error`. It keeps its Sandboxes, because they are the evidence of the fault.
+The operator removes them after the same idle timeout. A run that a Machine fails, with `fail()`, ends with the status
+`failed`, and the Orchestrator deletes its Sandboxes as `CANCEL` does: the Machine chose that end
+([ADR-0073](../reference/decisions/0073-a-machine-fails-a-run-with-fail-and-an-uncaught-failure-ends-it-failed.md)).
+
+`jr2 status` shows a cancelled run with the status `cancelled` and the state that it was in.
+
+## One week of history
+
+jr2 keeps the row of a run that ended for 7 days after its last write. This is true for each status that is not live:
+`done`, `failed`, `error`, `cancelled`, `lost` and `drifted`. The Orchestrator deletes the older rows when it starts,
+and one time each hour after that. A live run stays, whatever its age. A run that waits at a Gate for a month is still
+there.
+
+You cannot change the 7 days. jr2 is not the record of the results of a run. A Machine sends its results where they
+belong, such as a pull request. `jr2 status` on a run that ended is for debugging. After the 7 days, `jr2 status` says
+that the run does not exist, and a prefix of its id no longer finds it.
+
+The week of a drifted run starts at the restart or the publish that refused it, not at its last transition.
+
+[`jr2 down`](../reference/cli.md#down) deletes the namespace of the Instance, and the store with it. The live runs go
+too.
+
+## Rules
+
+- The context of a Machine and the input of each actor hold only plain data, such as ids, counts and strings. A live
+  object, such as a server or a client, does not survive a restart.
+- The Orchestrator continues a run only under the Assembly that it started under, and only when its Machine has the same
+  shape.
+- The Orchestrator never continues a drifted run, and never moves it to a new Machine.
+- The Orchestrator never continues a cancelled run.
+- `CANCEL` is the only run event that `jr2 send` sends without `--gate`. `--keep` goes only with `CANCEL`.
+- jr2 keeps a run that ended for 7 days. The hourly sweep never deletes a live run.
+
+[Cancel a run](../guides/cancel-a-run.md) and [Watch a run](../guides/watch-a-run.md) give the commands and their
+output.
+
+## Read next
+
+- [Watch a run](../guides/watch-a-run.md)
+- [Cancel a run](../guides/cancel-a-run.md)
+- [Iterate on a Machine on kind](../guides/iterate-on-a-machine-on-kind.md)
+- [Effects](effects.md), [Workspaces](workspaces.md) and [Converge](converge.md)
+- Reference: [`jr2 send`](../reference/cli.md#send), [`jr2 status`](../reference/cli.md#status),
+  [`jr2 runs`](../reference/cli.md#runs), [`POST /runs/:runId/events`](../reference/http-api.md#post-runs-runid-events),
+  [`GET /runs/:runId`](../reference/http-api.md#get-runs-runid) and
+  [`RunStatus`](../reference/http-api.md#type-runstatus)
+- [ADR-0007](../reference/decisions/0007-durable-machine-state.md): the state of a Machine is durable
+- [ADR-0021](../reference/decisions/0021-workspace-continuity-is-a-lease-that-answers-back.md): the Lease and Continuity
+  of a Workspace
+- [ADR-0025](../reference/decisions/0025-cancel-ends-a-run-stop-parks-it.md): `CANCEL` ends a run, and a stop parks it
+- [ADR-0030](../reference/decisions/0030-a-snapshot-names-the-machine-it-was-written-under.md): a snapshot names the
+  Machine that wrote it
+- [ADR-0065](../reference/decisions/0065-a-persist-survives-a-crash-and-history-is-a-week.md): a save survives a crash,
+  and jr2 keeps a week of history
+- [ADR-0069](../reference/decisions/0069-the-orchestrator-is-a-kit-image-and-an-instance-publishes-its-assembly.md): a
+  run holds the Assembly that it started under