@jr2/orchestrator 0.4.1 → 0.6.0

package/docs/guides/cancel-a-run.md

@@ -0,0 +1,146 @@
+---
+title: Cancel a run
+description: End a live run with CANCEL, and delete its Sandboxes or keep them to look inside.
+---
+
+# Cancel a run
+
+End a live run with `CANCEL`, and delete its [Sandboxes](../reference/glossary.md#sandbox) or keep them to look inside.
+
+In the commands on this page, replace these values:
+
+- `<run id>`: the id of the run. A prefix of the id, 4 characters or more, also works with `jr2`. `jr2 run` prints the
+  id in the line `run <run id> (<workflow>) — attached`.
+- `<namespace>`: the namespace of your [Instance](../reference/glossary.md#instance). It is the
+  [`name`](../reference/config.md#name) in `jr2.config.ts`. When `jr2.config.ts` sets no `name`, it is the name of the
+  Instance folder.
+- `<context>`: the kubectl context that `jr2` uses. It is the current kubectl context, or the value of `--context`.
+
+Give each `jr2` command in the Instance folder.
+
+## Cancel the run
+
+```sh
+jr2 send <run id> --event CANCEL
+```
+
+```text
+→ context <context> / namespace <namespace>
+sent CANCEL to 0a90c659-a2f4-44ed-ae03-ca711ca30ce4
+```
+
+`jr2` prints the full run id, also when you give a prefix. Your run id is different. Then these things occur:
+
+- The run ends with the status `cancelled`. It does not start again when the
+  [Orchestrator](../reference/glossary.md#orchestrator) restarts.
+- The [Turn](../reference/glossary.md#turn) of each [Agent](../reference/glossary.md#agent) in the run ends.
+- The Orchestrator deletes each Sandbox of the run. When a Sandbox goes, a commit that you did not push goes with it.
+- A run that a `dispatch` [Effect](../reference/glossary.md#effect) of this run started does not end. It is a run of its
+  own. Cancel it with its own run id. `jr2 status <run id>` lists it in a `dispatched:` line. See
+  [Start a run from a run](start-a-run-from-a-run.md).
+
+A `jr2 run` that is attached to the run prints `→ cancelled`, then the state that the run was in. Then it prints the
+status of the run as JSON and exits with the code 0.
+
+## Keep the Sandboxes
+
+To look inside the Sandboxes after the run ends, add `--keep`:
+
+```sh
+jr2 send <run id> --event CANCEL --keep
+```
+
+```text
+→ context <context> / namespace <namespace>
+sent CANCEL to 723cf388-c1cf-4b42-a628-8b7715754ca1; its Workspaces are kept
+```
+
+The run ends as before, but the Orchestrator does not delete its Sandboxes. To open a shell in one, see
+[Look inside a Sandbox](look-inside-a-sandbox.md). A [Workspace](../reference/glossary.md#workspace) in
+[Placing](../reference/glossary.md#placing) deletes its Sandbox at once, also with `--keep`. Its Sandbox has no node
+yet, so it has nothing to look inside.
+
+Nothing renews the [Lease](../reference/glossary.md#lease) of a Workspace that you keep. The
+[operator](../concepts/architecture.md#the-operator) deletes its Sandbox 30 minutes after the last renewal. To delete
+the Sandbox before then, find its name. The label `jr2.dev/run` takes the full run id that `jr2 send` printed:
+
+```sh
+kubectl -n <namespace> get sandboxes -l jr2.dev/run=<run id>
+```
+
+[Look inside a Sandbox](look-inside-a-sandbox.md#find-the-sandbox) shows the output of this command. Then give the name
+to this command:
+
+```sh
+kubectl -n <namespace> delete sandbox <sandbox>
+```
+
+```text
+sandbox.core.jr2.dev "ws-723cf388-x-12-1k4kghb" deleted from <namespace> namespace
+```
+
+Your Sandbox name is different.
+
+## Read how the run ended
+
+```sh
+jr2 status <run id>
+```
+
+```text
+→ context <context> / namespace <namespace>
+{"runId":"0a90c659-a2f4-44ed-ae03-ca711ca30ce4","workflow":"task","instanceId":"19b49b58-2ea4-4d79-b100-49aad6e12013","status":"cancelled","value":"running",…}
+```
+
+`jr2 status` prints the status of the run as one line of JSON. Its `status` is `cancelled`, and its `value` is the state
+that the run was in when you cancelled it.
+
+## Cancel from HTTP
+
+A client that is not the CLI sends `CANCEL` to
+[`POST /runs/:runId/events`](../reference/http-api.md#post-runs-runid-events), with the Instance token. The path takes
+the full run id, not a prefix. To get the Instance token, see
+[Open the Console](open-the-console.md#get-the-instance-token). To keep the Sandboxes, add `"keep": true` to the body.
+This example reaches the Orchestrator through the port forward in [Open the Console](open-the-console.md):
+
+```sh
+curl -X POST http://localhost:4000/runs/<run id>/events \
+  -H "authorization: Bearer <Instance token>" \
+  -d '{"type":"CANCEL","keep":true}'
+```
+
+```text
+{"ok":true}
+```
+
+## Rules
+
+- `--keep` goes only with `--event CANCEL`. With `--gate` or with a different event, `jr2 send` exits with the code 2:
+  `jr2 send: --keep goes with --event CANCEL alone (it keeps a cancelled run's Workspaces)`
+- Without `--gate`, `CANCEL` is the only event. Each other event exits with the code 2:
+  `jr2 send: unknown event "<event>" (run control accepts: CANCEL; workflow events need --gate)`. To send an event of
+  the [Workflow](../reference/glossary.md#workflow) to a [Gate](../reference/glossary.md#gate), see
+  [Answer a Gate](answer-a-gate.md).
+- A prefix of fewer than 4 characters exits with the code 2: `jr2: run id "<run id>" is too short (need 4+ characters)`
+- A prefix that finds no run exits with the code 1: `no run "<run id>"`. A prefix that finds two or more runs exits with
+  the code 1, and lists them: `error: run id "<run id>" is ambiguous`
+- `CANCEL` changes only a live run. A run that has already ended does not change, and `jr2 send` still prints
+  `sent CANCEL to <run id>`. A full id of no run also prints `sent CANCEL to <run id>`, because `jr2 send` does not look
+  up a full id.
+- A run that faults is not live, so `CANCEL` does not delete its Sandboxes. The operator deletes them 30 minutes after
+  the last renewal of their Lease.
+- A cancelled run does not continue. To do the work again, start a new run.
+- [`POST /runs/:runId/events`](../reference/http-api.md#post-runs-runid-events) refuses a body with a different `type`
+  with the status 400: `unknown event type "<type>" (accepts: CANCEL)`
+- It refuses a `keep` that is not `true` or `false` with the status 400: `"keep" must be a boolean`
+- It refuses a request with no valid token with the status 401: `unauthorized`. It refuses a Sandbox token with the
+  status 403: `run state and control need the Instance token — a Sandbox token has neither`
+- A path with a prefix of the run id, or with an id of no run, gets `{"ok":true}` with the status 200. Nothing changes.
+
+## Read next
+
+- [Durable runs](../concepts/durable-runs.md)
+- [ADR-0025](../reference/decisions/0025-cancel-ends-a-run-stop-parks-it.md): why CANCEL ends a run and deletes its
+  Sandboxes.
+- [`jr2 send`](../reference/cli.md#send)
+- [Look inside a Sandbox](look-inside-a-sandbox.md)

package/docs/guides/clean-up.md

@@ -0,0 +1,206 @@
+---
+title: Clean up
+description: Remove old images and unused Repo resources with jr2 gc, and remove an Instance with jr2 down.
+---
+
+# Clean up
+
+Remove the images and the [Repo](../reference/glossary.md#repo) resources that nothing uses, or remove an
+[Instance](../reference/glossary.md#instance) from its cluster.
+
+The examples on this page use the [Get started](../get-started/install.md) names: the Instance `my-instance` and the
+kubectl context `kind-jr2`. Your output shows your names, sizes and counts.
+
+## Know what removes images
+
+You seldom need a command to remove images. [`jr2 up`](../reference/cli.md#up) removes the images that nothing uses at
+the end of each converge that succeeds. [`jr2 down`](../reference/cli.md#down) removes them after it deletes the
+namespace. [`jr2 gc`](../reference/cli.md#gc) does the same work when you ask for it, for example when your disk is
+full.
+
+jr2 removes only an image that it built and labeled, and only when no live root in the cluster names it. It removes the
+image from your computer. On a kind cluster, it also removes it from each node. It does not remove an image from a
+registry. [Converge](../concepts/converge.md#clean-up) explains the live roots.
+
+## See what jr2 gc removes
+
+Run `jr2 gc` with `--dry-run` first. It removes nothing, and it lists each image and each Repo resource that it would
+remove:
+
+```sh
+jr2 gc --dry-run
+```
+
+```text
+jr2 gc — images on kind-jr2 that no live root names (dry run: nothing is removed)
+would sweep 1 image(s) (336.5 MB)
+  jr2-sandbox-my-instance-default:6389bdf72d50-amd64
+repos: would sweep nothing — every Repo resource is bound by a Machine or was attached within 7d
+```
+
+Here, the image is an old Sandbox Image that the last `jr2 up` kept on the kind node.
+
+`jr2 gc` reads the cluster of the current kubectl context. It does not read an Instance folder, so you can run it from
+any folder. It reads each Instance on the cluster, so it has no `-n` option. To use a different cluster, add
+`--context <context>`.
+
+## Remove the images and the Repo resources
+
+```sh
+jr2 gc
+```
+
+```text
+jr2 gc — images on kind-jr2 that no live root names
+swept 1 image(s) (336.5 MB)
+repos: swept nothing — every Repo resource is bound by a Machine or was attached within 7d
+```
+
+In this output, `jr2 gc` removed no Repo resource. `jr2 gc` does not ask you to confirm. It removes a Repo resource when
+these two conditions are true:
+
+- No registered [Machine](../reference/glossary.md#machine) binds the Repo.
+- No run attached the Repo, and nothing made it, in the last 7 days.
+
+After jr2 removes a Repo resource, the Repo cache on each node can remove its copy of the Repo. It removes the copy when
+nothing on the node uses it. `--repo-ttl` changes the time limit of the second condition. For example, `--repo-ttl 12h`
+removes a Repo resource that no Machine binds and that no run attached in the last 12 hours. `--repo-ttl 0` removes each
+Repo resource that no Machine binds. The last line then gives the count and the time limit:
+`repos: swept 1 Repo resource(s) no Machine binds and no run attached within 0`.
+
+## Remove an Instance from the cluster
+
+Run `jr2 down` in the Instance folder:
+
+```sh
+jr2 down
+```
+
+```text
+remove instance "my-instance" from context kind-jr2 (delete namespace "my-instance")? [y/N] y
+deleting namespace "my-instance" (runs, store, and Sandboxes go with it)
+swept 2 image(s) (317.0 MB)
+removed
+```
+
+`jr2 down` asks you to confirm, unless you add `--yes`. Type `y`. Then it deletes the namespace of the Instance and
+waits until the namespace is gone. These things go with the namespace:
+
+- The [Orchestrator](../reference/glossary.md#orchestrator) and the runs, with the store of the runs.
+- Each [Sandbox](../reference/glossary.md#sandbox). A commit that is not pushed is lost with its Sandbox.
+- The [Instance Harness](../reference/glossary.md#instance-harness) and the conversations that it keeps.
+- Each Secret in the namespace. This includes the Instance token and the Secrets that you made with `kubectl`, for
+  example for [`harness.envFrom`](../reference/config.md#harness-envfrom), for a `valueFrom` in
+  [`secrets`](../reference/config.md#secrets) and for the git ssh key.
+
+Then `jr2 down` removes the images that nothing uses now. The Instance folder does not change.
+
+To put the Instance on the cluster again, make your Secrets again first. Then run `jr2 up`. When `jr2 up` makes a new
+deploy key for git, it is a new key. Register its public key again. See
+[Give the cluster git credentials](give-the-cluster-git-credentials.md).
+
+`jr2 down` does not remove the copies of the Repos on each node. They are in `/var/lib/jr2/<namespace>/repos` on the
+node, and nothing uses them. On kind, they are in the node container, and they go when you delete the cluster.
+
+To remove the Instance in CI, add `--yes`. To remove it from a different namespace or cluster, add `-n <namespace>` or
+`--context <context>`.
+
+## Remove the operator too
+
+The operator is in the namespace `jr2-system`. All the Instances on a cluster share it. When you remove the last
+Instance, run `jr2 down --all` in place of `jr2 down`:
+
+```sh
+jr2 down --all
+```
+
+```text
+remove instance "my-instance" from context kind-jr2 (delete namespace "my-instance" AND the per-cluster operator)? [y/N] y
+deleting namespace "my-instance" (runs, store, and Sandboxes go with it)
+uninstalling the operator (jr2-system)
+swept 2 image(s) (259.1 MB)
+removed
+```
+
+> **Danger**
+>
+> Use `--all` only when no other Instance is on the cluster. `--all` deletes the Sandbox and Repo resource types. Thus the
+> Sandboxes and the Repo resources of each Instance on the cluster go with them.
+
+`jr2 down --all` removes the operator only while the namespace of the Instance is on the cluster. After a plain
+`jr2 down`, it refuses with `not deployed here …`. To remove the operator then, run `jr2 up` again. Then run
+`jr2 down --all`. On kind, you can delete the cluster.
+
+`jr2 down --all` does not remove the two PriorityClasses that `jr2 up` adds, `jr2-control` and `jr2-sandbox`. To remove
+them, run this command:
+
+```sh
+kubectl delete priorityclass jr2-control jr2-sandbox --ignore-not-found
+```
+
+When [`priorityClasses`](../reference/config.md#priorityclasses) names your own class for `control` or `sandbox`,
+`jr2 up` does not add `jr2-control` or `jr2-sandbox` for that tier.
+
+## Remove the kind cluster
+
+Run `jr2 down` in each Instance folder. For the last one, add `--all`. Then delete the kind cluster:
+
+```sh
+jr2 down --all
+kind delete cluster --name jr2
+```
+
+`jr2 down` removes the images of its Instance from your computer. `kind delete cluster` cannot do this, because the
+images are in your docker, not in the cluster. The images that your Dockerfiles name in `FROM` stay, and so does the
+build cache of docker.
+
+## Rules
+
+- `jr2 down` runs only in an Instance folder. In a different folder, it exits with the code 1:
+  `error: not inside a jr2 instance — no jr2.config.ts found walking up from cwd`
+- `jr2 down` with no kubectl context exits with the code 1: `error: no kube context — nothing to remove from`
+- `jr2 down` refuses a namespace that holds no Instance, and exits with the code 1:
+  `not deployed here — namespace "<namespace>" on <context> holds no jr2 instance`. Check the context and the namespace.
+  `-n` and `--context` change them. `jr2 down --all` also refuses this way after a plain `jr2 down`, and the operator
+  stays.
+- `jr2 down` refuses a namespace that belongs to a different Instance, and exits with the code 1:
+  `refusing: namespace "<namespace>" on <context> belongs to another instance ("<owner>")`
+- When you do not type `y` or `yes`, `jr2 down` changes nothing, and exits with the code 1:
+  `aborted — nothing was changed`
+- When no terminal is attached, `jr2 down` cannot ask, so it changes nothing. It prints
+  `<question> — not a tty; pass --yes to proceed non-interactively`, then `aborted — nothing was changed`. It exits with
+  the code 1. Add `--yes`.
+- `jr2 down` refuses when the Instance and the [jr2 CLI](../reference/glossary.md#jr2-cli) do not use the same
+  [Kit version](../reference/glossary.md#kit-version). The refusal starts with
+  `error: Kit version mismatch — an Instance has one (ADR-0056):`. See
+  [Fix a failed converge](fix-a-failed-converge.md#rules).
+- When the namespace is gone but the image sweep fails, `jr2 down` exits with the code 0. It prints:
+  `image sweep failed (<error>) — the instance is still removed`. Run `jr2 gc` later.
+- `jr2 gc` with no kubectl context exits with the code 1:
+  ``error: no kube context — `jr2 gc` decides what is garbage by asking a cluster what it still needs``
+- `jr2 gc` refuses a `--repo-ttl` that is not a number with `d`, `h` or `m`, or `0`. It exits with the code 1 before it
+  reads the cluster: `error: --repo-ttl "<ttl>" is not a TTL — use <n>d, <n>h, or <n>m (or 0 for now)`
+- When `jr2 gc` cannot read each live root, it removes nothing, and exits with the code 1:
+  `error: the roots could not be read (<error>)`, then
+  `  nothing was swept — a keep set missing one root would delete images the cluster still needs`. The error in the
+  parentheses can come from kubectl, from docker or from a kind node. Fix that cause, then run `jr2 gc` again.
+- When `jr2 gc` cannot read the Repo resources, it exits with the code 1:
+  `error: the Repo resources could not be read (<error>)`
+- On a kind node, jr2 removes a whole image. Thus it keeps a tag whose image also has a tag that a live root names. It
+  prints: `kept <count> tag(s) — their image id also carries a tag a live root names: <tags>`. On your computer, jr2
+  removes each tag that no live root names.
+- When docker or a node refuses to remove an image, jr2 prints: `failed to remove <count> image(s): <images>`
+- When `jr2 up` cannot remove the old images, the converge still succeeds. It prints:
+  `images: sweep skipped (<error>) — the converge stands`. Run `jr2 gc` later.
+
+## Read next
+
+- [Converge](../concepts/converge.md)
+- [ADR-0039](../reference/decisions/0039-image-garbage-collects-by-reachability.md): why jr2 removes an image when no
+  live root names it.
+- [ADR-0051](../reference/decisions/0051-a-repo-is-a-slot-on-the-workspace-and-a-cache-on-the-node.md): why `jr2 gc`
+  removes a Repo resource when no Machine binds it and no run attached it in the last 7 days.
+- [ADR-0019](../reference/decisions/0019-one-converging-command-against-the-current-context.md): why `jr2 down` asks you
+  to confirm.
+- [`jr2 gc`](../reference/cli.md#gc), [`jr2 down`](../reference/cli.md#down)
+- [Cancel a run](cancel-a-run.md)