@jr2/orchestrator 0.4.1 → 0.6.0

package/docs/reference/decisions/0018-instance-agents-are-definitions-jr2-assembles-the-harness.md

@@ -0,0 +1,111 @@
+# The instance authors Agent definitions; jr2 assembles the Harness
+
+Building the first real Harness (the jr-parity exercise, 2026-07-18) landed a **full harness-framework project** in the
+instance — package.json, runtime config, Dockerfile, per-agent source (under flue, the embedded framework since retired
+— [ADR-0027](0027-the-harness-is-jr2s-own-server-flue-retires-the-wire-stays.md)). That put jr2 mechanism in the user's
+hands: the [ADR-0013](0013-the-agent-reaches-its-machine-through-a-container-it-cannot-read.md) leash (read the Menu per
+Submission — forget it and the Agent is silently mute, the same footgun class ADR-0016 removed from workflows), the
+ADR-0004/0012 sandbox geography (working tools execute in the Harness container against `/work`), the dependency pin
+that must match the Orchestrator's wire, and the image contracts the operator imposes (numeric uid, `:8080` = Ready, git
+present). The only content a user genuinely owns is the persona: model + instructions (+ someday tools/skills).
+GLOSSARY.md already promised the split — an Agent is a definition jr2 maps into the Harness, and the Instance folder
+holds only what a Machine cannot carry (ADR-0049/0050: the definition is one of the things it can).
+
+## Decision
+
+- **An Agent is a plain-data definition carried by a Machine as an actor slot** —
+  `actors: { <name>: agent({ model, instructions, … }) }`, slot key = Agent name, invoked as `src: "<name>"` (ADR-0049).
+  The definition may be written inline or imported from any module. Serializable persona data only; **the instance never
+  imports the Harness runtime**. `model` is **required**: every Agent is independently valid, and definitions are the
+  only models `jr2 up` can preflight — it collects them by walking the registered Machines. (The first cut
+  filename-discovered `agents/<name>.ts`, mirroring `workflows/`, and a second put them in `jr2.config.ts`; both were
+  Instance rosters, and a roster cannot hold two Machines' `coder`s.)
+- **The Harness image is stock and published** — `jr2-harness:<kitversion>`, one release train with the kit (like the
+  operator image, ADR-0019). An instance builds **no** Harness image. The image is `@jr2/harness`, jr2's own server
+  (ADR-0027), honoring the operator's image contracts.
+- **The definition rides the Turn, never the image.** The admission body carries the slot's definition (ADR-0049); the
+  server validates it loudly and re-reads model/instructions per Submission, with the Turn's Frame (`prompt`, `cwd`) and
+  Dials beside it on the admit body (ADR-0057). No image is built anywhere for a definition edit. (A ConfigMap roster
+  read at boot — `JR2_AGENTS_JSON` — was the mechanism between the codegen-at-boot cut and this one; it retired because
+  a Machine edit already publishes the Assembly again, so the roster bought nothing a Machine-carried definition loses.)
+- **The kit's server assembly carries the mechanism**: the Menu leash, rooting the Working tools at the Frame's cwd,
+  tool assembly — existing only in kit code, unforgettable by construction (symmetric with ADR-0016 making
+  endpoint/sandbox threading unrepresentable in workflows).
+- **`harness` is the agent-runtime section of `jr2.config.ts`** — it declares what the instance can **reach**, never
+  which model to use: a custom provider (`{ api, baseUrl }` plus token limits — `contextWindow`/`maxTokens`,
+  provider-level and per-model, since a custom id has no catalog entry and unset limits resolve to 0, starving
+  auto-compaction — e.g. an OpenAI-compatible vLLM endpoint), the gateways pi's catalog providers send their calls to
+  (`catalog`, below), and any other env Agents need; the credentials Agents use are the Held secrets of the top-level
+  `secrets` list (below). It moved out of `sandbox` deliberately: `sandbox` is pod-transport (it still _carries_ this
+  env to the Harness container), but model concerns are Harness semantics and users configure them here. `jr2 up`
+  preflights every model the definitions name, from inside the cluster (ADR-0019). A model of pi's catalog whose
+  provider no pod holds a key for fails the preflight, unless a credential in `harness.env` (an AWS role for Bedrock,
+  say) reaches it. An instance-wide default model was tried and removed: it was reached through `.env` (`JR2_MODEL`),
+  which put a design decision in the file reserved for deployment-varying values (ADR-0019) — and split one fact in two,
+  since the endpoint's per-model token limits were already committed in `jr2.config.ts` keyed by the very model id
+  `.env` was choosing. The honest split: **`harness` declares reach; the definition makes the choice.**
+- **Credentials are held, not handed over**
+  ([ADR-0059](0059-a-harness-holds-stand-ins-and-the-custodian-holds-the-keys.md)). The Agent executes code in the
+  Harness container, so a key there is a key it can take away. The top-level `secrets` list declares each credential
+  ([ADR-0068](0068-an-effector-holds-an-effects-secrets-and-signs-its-wire.md)). A model key is a Held secret in the
+  pods whose Agents name a model of a provider that reads it; `jr2 up` gives its value to the Custodian of those pods —
+  and, where an Effect also names it, to that Effect's Effector — and the Harness container's env holds the key's
+  Stand-in. `harness.provider.apiKey` is one implicitly: the held secret `JR2_PROVIDER_API_KEY`, bound to `baseUrl`'s
+  host, so it leaves the `jr2-harness-env` Secret and `baseUrl` must be `https://` when a key is set. A known model key
+  in `harness.env`, or in an `envFrom` Secret, is refused.
+- **A catalog provider can be moved to a gateway, and stays a catalog provider.** `harness.catalog.<id>.baseUrl`
+  replaces the base URL of every model of one pi catalog provider — a LiteLLM gateway in front of Bedrock, for
+  `anthropic/…`. The catalog keeps its limits, cost and thinking levels; only the endpoint moves. It is reach, not a
+  model choice: it says nothing about which model an Agent names.
+
+## Dials: a workflow may set two of a definition's fields for one Turn
+
+One definition value may be carried by several Machines, so one persona legitimately runs at different settings in
+different workflows — a reviewer on a one-line diff and the same reviewer on an architecture change want identical
+instructions and different effort. Spread-composition (below) answers this with persona×tier files where "heavy" is a
+knob wearing a persona's filename, so `agentRun` takes two optional **dials**, `model` and `thinkingLevel`, layered over
+the definition per Submission. A model can lack a level (Claude Sonnet 5.5 has no `off`). The Harness does not refuse it
+at admission: the Turn runs, and the printed conversation names the level that the model ran at
+([ADR-0023](0023-the-harness-prints-the-conversation.md)).
+
+The line that keeps this from becoming "re-specify the definition at the call site": **identity vs. dial.** Identity —
+`instructions`, `access` — is definition-only: a call site that rewrote it would make the Agent's name a lie, and
+`access` carries [ADR-0028](0028-what-an-agent-may-do-to-the-workspace-is-part-of-its-definition.md)'s containment claim
+that a read-only reviewer _cannot_ write, which per-invocation escalation would void. **ADR-0028 is therefore untouched
+by the dials.** Dials say only how hard to run: it is still the coder, it is the coder running hot. What the Turn is
+about and where it works — `prompt`, `cwd` — is neither identity nor a dial but the Turn's **Frame**
+([ADR-0057](0057-a-turn-is-its-frame-its-dials-and-whether-it-continues.md)): per Turn by nature, since a worktree path
+exists only once a run does, and no more a property of the persona than the prompt is.
+
+Where a model is checked: two seats, each where the knowledge is. At **converge**, `jr2 up` walks the registered
+Machines for the definitions they carry (ADR-0049) and checks a configured provider once per distinct model they name.
+At **admission**, the Harness resolves the definition it was handed against its registry and 400s the invoke as its
+state is entered, rather than settling the Submission `failed` mid-run. The pod-boot seat retired with the roster it
+read: the Harness no longer knows an Agent before a Turn brings it one. A dial has only ever had the admission seat — a
+call-site model cannot be checked earlier, because an invoke's `input` is a function and is not statically recoverable.
+Definitions keep converge-time safety; overrides pay the later check for the flexibility.
+
+## Considered options
+
+- **Thin wrapper** (a jr2 identity imported inside a user-owned harness project). Rejected: shrinks the per-agent file
+  but leaves the dependency pin, the runtime config, and the Dockerfile contracts in user space — the parts that break
+  silently when they drift from the Orchestrator's wire and the operator's pod.
+- **Instance owns the harness project** (the first cut). Rejected: violates the documented Instance/Agent split, and
+  every new agent re-copies the leash.
+- **Per-instance Harness image built by a `jr2 harness build`** (the second cut: jr2 generates the project under
+  `.jr2/harness/`, the user docker-builds it). Rejected once definitions were plain data: the image content was 100%
+  kit-owned mechanism plus a JSON's worth of persona — so the persona moves at runtime and the image joins the published
+  release train. This also deleted a build + `kind load` step from every agent edit.
+
+## Consequences
+
+- The definition contract is data-only: what an Agent may do to the Workspace is part of it (`access`, ADR-0028); custom
+  tools/skills/subagents have no seat yet. When someone needs them, the contract grows deliberately — there is no eject
+  hatch to a foreign harness project (ADR-0027 deleted it).
+- Because definitions are plain data, stock ones compose with no API: a Machine carries `agent(coder)`, and a consumer
+  retunes it with `customize(machine, { agents: { coder: { model: "…" } } })` (ADR-0049, ADR-0009).
+- An invalid definition fails at admission, loudly, naming the slot: a model no registry resolves, a malformed field. A
+  Machine with no Agent slots is valid — a `workspace()` body parking a Sandbox (ADR-0012) invokes none. Readiness is
+  binding `:8080`, with no boot build ahead of it (ADR-0027).
+- The kit owns version compatibility: the Harness runtime and its exact pins are kit concerns (ADR-0027) — bumping them
+  is a kit change, never an instance chore.

package/docs/reference/decisions/0019-one-converging-command-against-the-current-context.md

@@ -0,0 +1,287 @@
+# The operational surface is one converging command against the current kube context
+
+Bringing the first real workflow to a cluster (2026-07-18) took four command surfaces (`jr2`, `just`, `kind`,
+`kubectl`), three long-running shells, and a runbook of ordered steps whose footguns the CLI absorbed none of: kind
+cluster names were host-global and instance-bound, images had to be re-`kind load`-ed after every cluster recreate, a
+missing `.jr2/cluster.json` surfaced as a provision-time routing error, and a missing Secret parked pods in
+`CreateContainerConfigError`. Root cause: the **local/remote split**. `jr2 dev` ran the Orchestrator on the host while
+Sandboxes ran in the cluster, which forced two host↔cluster seams (the pod→host callback route and the `repos/` hostPath
+baked at cluster creation) — and every footgun above is one of those seams leaking. If a runbook is needed to use `jr2`,
+the interface has failed.
+
+## Decision
+
+**The Orchestrator always runs in-cluster.** There is no host dev mode. `jr2` targets whatever cluster the current
+`kubectl` context points at (the argo/cilium model) — which may be a local kind cluster, but nothing is kind-special: no
+baked mounts, no recorded pod→host route, no `.jr2/dev.json` / `.jr2/cluster.json`. `jr2 dev`, `jr2 cluster up|down`,
+and `jr2 harness build` are removed.
+
+**`jr2 up` is the one command.** It idempotently converges the target namespace to match the instance — every layer,
+loudly narrated, safe to re-run:
+
+- **Operator** (per-cluster, shared): CRD + Deployment applied from manifests shipped _inside the installed npm
+  package_. The image ref follows [ADR-0038](0038-jr2-up-builds-every-image-it-deploys.md): in a kit checkout, `up`
+  builds the operator image and content-addresses it; installed from npm, the ref is the published `<kitversion>` tag at
+  the canonical home, re-homed by `kitRegistry`
+  ([ADR-0044](0044-kit-images-live-at-a-canonical-home-a-self-host-mirrors-it.md)). `up` upgrades an older deployed
+  operator and **never downgrades** — if another instance upgraded it past this kit's pin, warn and leave it (CRD
+  versioning is the skew contract). `operator.manage: false` skips the layer to run the controller loop in the
+  foreground (kit development).
+- **Orchestrator** (per-instance): a Kit image at the Kit version, resolved like the operator's, plus the instance's
+  **Assembly** — its `workflows/`, config and deps, with the kit's packages stripped — staged, sealed, and
+  **content-addressed by the staged tree**, then published to the running Orchestrator over its API
+  ([ADR-0069](0069-the-orchestrator-is-a-kit-image-and-an-instance-publishes-its-assembly.md)). A kit upgrade rolls the
+  image; a Machine edit publishes an Assembly and rolls nothing. Hashing the instance _folder_ instead — where deps
+  appear only as version ranges — is what let `up` skip work it needed. `--force` publishes against an unchanged hash.
+- **Harness**: no per-instance Harness image (ADR-0018). The Harness is a Kit image, resolved like the operator's
+  (checkout-built or published, ADR-0038/0044), and the Custodian beside it in every Harness pod runs a Pinned image
+  ([ADR-0059](0059-a-harness-holds-stand-ins-and-the-custodian-holds-the-keys.md)); the Harness reaches each Sandbox as
+  an `/opt/jr2` volume injected at pod time, over a Sandbox Image the instance builds or brings
+  ([ADR-0037](0037-an-instance-builds-its-sandbox-images-jr2-injects-the-harness.md)). Each Turn carries its Agent's
+  definition to the Harness in the admission (ADR-0049); `up` delivers no roster.
+- **Secrets**: values declared in config (which may read `process.env`, populated from the uncommitted `.env`) are
+  materialized into an instance-owned Secret, by server-side apply, so that no value is in a
+  `last-applied-configuration` annotation
+  ([ADR-0059](0059-a-harness-holds-stand-ins-and-the-custodian-holds-the-keys.md)). Referenced-but-unmanaged Secrets
+  (`envFrom` refs, git credentials) are **preflighted**: `up` fails naming the missing Secret with the exact creation
+  hint — converting the `CreateContainerConfigError` hang into an immediate error. Sealed Secrets/External Secrets ride
+  this seam untouched: jr2's contract is "a Secret named X exists", however it got there. `up` also refuses a held
+  secret's name that is a key of an `envFrom` Secret, and names the Secret, the key and the held secret (ADR-0059, R14).
+  A credential the Harness uses is a held secret
+  ([ADR-0059](0059-a-harness-holds-stand-ins-and-the-custodian-holds-the-keys.md)): `up` writes a literal value into
+  `jr2-held-secrets-<pod>` for each pod that names it, which only that pod's Custodian mounts, and checks that a
+  referenced Secret and key exist. Like the signing key, the held-secret CA (`jr2-held-ca`) is kept across converges —
+  minted on first need, never rotated by `up` — because a live pod trusts it.
+- **Repos**: `up`'s walk of the registered Machines collects every bound Repo Slot (ADR-0049, ADR-0051) — to refuse an
+  open slot nobody bound, and to see which urls need a key. The Orchestrator creates one `Repo` resource per url at
+  boot, and the operator's cache agent clones each onto the nodes that need it (ADR-0004). There is no `repos` config
+  list and no host `repos/` directory. Private repos: `git.credentials` matches each url to a token env var or an ssh
+  Secret by longest prefix (ADR-0051); the scaffold's wildcard entry makes `JR2_GIT_TOKEN` from `.env` the default path
+  for https; for ssh URLs whose entry names a Secret that does not exist, `up` asks where the key comes from — a fresh
+  in-cluster deploy keypair (the recommended default, public key printed to register), a local key, or one pasted on
+  stdin ([ADR-0047](0047-the-git-ssh-key-source-is-the-users-choice.md)) — declining bails. A personal key never enters
+  a cluster silently: only by that explicit, warned choice.
+- **Landlock**: `up` refuses to converge while a Sandbox node lacks Landlock. It reads the pods of the Repo cache
+  agent's DaemonSet, which probe Landlock on each Sandbox node, within the rollout wait it already makes for the cache.
+  It is an ordinary converge failure: it names each node whose agent reports none, the probe's reason, and the fix (keep
+  the node out with `sandbox.nodeSelector` or a taint, ADR-0052, or enable Landlock); a run after the fix converges
+  ([ADR-0077](0077-jr2-requires-landlock-on-every-sandbox-node.md)).
+- **Provider preflight**: `up` checks each provider that a carried model names. It checks from inside the cluster, with
+  one trivial tool-call completion. An endpoint that the pods cannot reach, a wrong or missing key, or a vLLM without
+  `--enable-auto-tool-choice` then fails at converge time, not as `agent.fault` in a run. Reachability is the user's
+  concern (`localhost` never works from a pod; a LAN address does on kind). The check must take the path that the
+  Harness takes. If it does not, a pass means nothing.
+  - **One path for every provider.** `up` checks the custom provider (`harness.provider`, ADR-0018) and a provider of
+    pi's catalog (for example `anthropic/…`) in the same way. A key of either is a held secret, and only a Custodian can
+    use it ([ADR-0059](0059-a-harness-holds-stand-ins-and-the-custodian-holds-the-keys.md)). So no key enters a pod
+    spec, an argv or a script. The check calls the custom provider as a Turn calls it: it does not force the tool,
+    because a vLLM without a tool parser answers a forced tool and then fails every Turn. When the custom provider
+    answers twice with no tool call, the check fails. `up` refuses a `harness.provider.api` that the Harness does not
+    speak before it builds or applies anything: the list is the Harness's own (`CUSTOM_APIS`, held equal to
+    `CUSTOM_PROVIDER_APIS` in `@jr2/orchestrator` by a test). When the check container cannot build its registry from
+    the config, each model's result says why, in words.
+  - **The check pod.** It is a Harness pod with no Harness in it. It has a Custodian and the stock Harness image's
+    `provider-preflight.ts`. The function that composes the Custodian of every Harness pod composes this Custodian too.
+    Its token is a value that the Orchestrator never signed. Its Stand-ins are secret and its own: `jr2 up` mints them
+    with the signing key, as it does for the Instance Harness
+    ([ADR-0059](0059-a-harness-holds-stand-ins-and-the-custodian-holds-the-keys.md)). The check pod runs no Workspace
+    code, so it may hold them in env. `provider-preflight.ts` gets the env of the Harness container, with the Stand-ins
+    in it: the harness config, `harness.env`, `harness.envFrom`, the Stand-ins, the proxy and the trust. pi resolves
+    each model and uses its api, as in a Turn. The `harness.catalog` gateway applies. The key never enters the check
+    container, its argv or its log.
+  - **Its own objects.** The check pod reads no object of the Instance. `up` writes copies from the values on the host:
+    - a ConfigMap with the Custodian's files, the program and the harness config;
+    - a ConfigMap with the trust bundles;
+    - a Secret with the token, the check pod's Stand-ins and the literal values of the held secrets;
+    - a Secret with the leaves;
+    - a Secret with the literal `harness.env` values.
+
+    A held secret from `valueFrom` stays a reference to the user's Secret. A literal value is thus in a second Secret
+    while the check pod lives. `up` deletes that Secret after the check.
+
+  - **What it checks.** It checks every carried model that a Harness pod runs. A typo, a retired model and a model that
+    the key cannot use are each per model. One check pod stands for one class of Harness pod. A class is a placement and
+    a held set:
+    - The placement: a Workspace's model runs on the Sandbox nodes (`config.sandbox`, ADR-0052). A `"none"` Agent's
+      model runs where the Instance Harness runs.
+    - The held set: a pod that holds a secret sends its HTTPS through its Custodian, and a tunnel carries a host that no
+      secret is bound to. A pod that holds no secret sends its HTTPS direct. So the check pod holds the held set of the
+      first pod of its class, and the keys that the other pods of the class hold. A class that holds nothing gets a
+      check pod that holds nothing.
+
+    When no pod of the class holds a key of the provider, the check pod holds none. Then pi refuses the call
+    (`not configured`), and the converge fails. A signing credential in `harness.env` reaches the check pod, as it
+    reaches a Turn. An id that pi does not know (a typo) fails, because pi resolves no model for it. An id that pi
+    knows, and for which jr2 knows no key name, is checked with no held key, as a Turn runs it. The check pod has the
+    priority class of the Sandbox tier, as the Harness pods have (ADR-0060).
+
+  - **The request.** The check's request shares these with a Turn's: the model, its api, its endpoint and key, the
+    completion limit (what pi-ai sends for a Turn), the retries, no forced tool, and the level of a definition with no
+    `thinkingLevel` (`off`: no `reasoning` is sent). It does not share these: it has no system prompt, one tool, no
+    session id, and a one-line prompt (one text part, as a Turn's user message has parts). These are limits of the
+    check:
+    - An endpoint whose context is too small for a Turn's prompt can pass the check.
+    - A definition with a `thinkingLevel` is checked at the default level, because `up` knows the models of the
+      definitions and not their levels.
+  - **The call.** The first call is the request that a Turn sends. So a pass always means that the Turn's request works.
+    Its answer decides:
+    - A tool call: the check passes.
+    - An error: the check fails, or it is busy (below). A forced tool could pass where a Turn fails: a vLLM behind a
+      `harness.catalog` gateway, with no `--enable-auto-tool-choice`, answers a named tool by guided decoding and
+      refuses the Turn's request.
+    - Text from the custom provider: a Turn there gets a nudge after text, so the check sends the same call once more. A
+      tool call then passes, an error decides as above, and text again fails, because that is how a vLLM with no tool
+      parser answers.
+    - Text from any other model: a warning. When the model is of pi's catalog, has `off`, is not moved to a gateway, and
+      its api has a form for a forced tool, a second call forces the tool. It does not change the verdict, only the
+      words of the warning: with a tool call, the model answered in text when the tool was not forced; with text or an
+      error, the model gave no tool call either way. Anthropic refuses a forced tool while the model thinks, so a model
+      with no `off` gets no second call.
+  - **Busy.** A provider that is busy after the retries of a Turn ([ADR-0064](0064-a-workspace-waits-for-capacity.md))
+    is a warning: it answered, and it took the key. Busy is a 429, a 529, an overload, a rate limit or
+    `ResourceExhausted`. A 503 alone is a failure, because a gateway with no backend and a server that still loads its
+    model answer 503, and a Turn there fails. The check waits at most 20 s for one `retry-after`. When a provider asks
+    for more, the call stops with its 429, and that is busy.
+  - **Time.** A call waits 120 s for its first event (text, thinking, a tool call or the end), and then 60 s for each
+    gap between two events. An Ollama endpoint loads the model on the first request, and a llama.cpp on a CPU reads the
+    prompt before it writes: both are slow to start, and then they stream. An endpoint that stops for a minute in the
+    middle of an answer is one that a Turn cannot use either. Each model has 360 s for all its calls: two calls, each
+    with 120 s to its first event and 60 s of the gaps that follow. Each model has its own 360 s, so one slow model does
+    not use the time of the others. Two literals set it, `PREFLIGHT_MODEL_BUDGET_SECONDS` in `preflight-line.ts` for the
+    check and `CHECK_MODEL_SECONDS` in `up.ts` for the pod, and a test holds them equal. A call or a model that runs out
+    fails. For the custom provider and a gateway, the error adds that a model that loads on first use can take longer.
+  - **Busy, when a call runs out.** The rule: the last status that any try of the call got was a 429 or a 529, and then
+    the call ran out. A later 200 or 503 clears it. pi-ai sends each retry as a new request. The check sees each try
+    through a `fetch` wrapper of its own. It sees Bedrock's tries through `onResponse`, because Bedrock's retries are
+    the AWS SDK's own (pi-ai's `maxRetries` does not reach them). Google's and Vertex's adapters refuse a `fetch` that
+    is not `globalThis.fetch`, so the check passes them none and does not replace the global one. For them only the
+    final error counts: a 429 followed by a retry that never answers fails the check, where for the other apis it is
+    busy. A second call that runs out does not change the verdict of the first.
+  - **The order.** The checks run before `up` writes an object that the Instance or the operator reads. Only these come
+    before them, because the check pod needs them: the namespace and its label, the PriorityClasses, and the images that
+    `up` builds and delivers. The operator (its CRDs, RBAC and Deployment, its label and its rollout), a deploy key and
+    the wait for its registration (ADR-0047), the trust bundles, the held objects and the instance List come after every
+    check passes.
+  - **When a check fails.** The converge stops. `up` applied only the items above and the check pods with their own
+    objects, and it deleted the check pods and their objects. It also deleted the leftovers of earlier runs (below). The
+    operator, the Orchestrator, its Assembly and every pod of the Instance continue to run as before. Nothing else is
+    deleted, and no Assembly is published.
+  - **The pods.** `up` runs the check pods in series, one at a time, and reports every failure in one error. When the
+    cluster refuses a check pod (a quota, a Pod Security level, a webhook), that check fails, and the other checks still
+    run. In a new namespace, `up` waits up to 10 s for the `default` ServiceAccount before it calls a refusal a refusal.
+    The check container writes a start line and a result line for each model. `up` reads its log while it runs, and
+    writes each result as it comes. When the pod stops before the end, the error names the model that the check was on,
+    and each model that it never reached. The name of each pod is unique to the run
+    (`jr2-provider-preflight-<run>-<n>`), so two runs in one namespace do not touch each other's pods. `up` deletes the
+    objects of each check after it. `up` waits up to 300 s for the check container to start (two cold image pulls), and
+    it writes each new reason that the pod gives for the wait. Then it waits 360 s for each model (the model's own
+    budget in the check), and 30 s more for the Node.js process, which runs under tini as the Harness does, to start. It
+    stops at once in these cases:
+    - A container cannot start: a crash loop, a missing Secret or key in the env (`CreateContainerConfigError`), or a
+      pull that the registry refuses (not found, manifest unknown, unauthorized, denied). A slow or limited pull, and a
+      `CreateContainerError`, can pass by themselves, so the wait goes on.
+    - A volume cannot mount: the Secret or the key of a held secret's `valueFrom` is missing. The kubelet says so in a
+      `FailedMount` event. It counts when the kubelet says it again (its count or its last time goes up), because the
+      first can be a race with the objects of the same apply.
+    - The Custodian restarts. The error then quotes the Custodian's refusal line, or the reason of its last end
+      (`OOMKilled`).
+  - **Leftovers.** `up` has no signal handler. If `up` stops (Ctrl-C), the kubelet stops a running pod at its
+    `activeDeadlineSeconds`. A pod that no node took has no start time, and that deadline never applies to it. So a
+    later `up` of the same Instance deletes these objects:
+    - each check pod that ended more than 10 minutes before: in that time, the run that made the pod can still read its
+      log;
+    - each check pod, in any phase, that is older than its `activeDeadlineSeconds` and 10 minutes;
+    - the objects of those pods, and each object of a check that is more than one hour old and whose pod is gone.
+
+    It never deletes the objects of another Instance.
+
+  - **Skipping it.** `jr2 up --skip-provider-preflight` runs no check pod. A check can fail when the model works, for
+    example when a model loads for longer than the check waits. That fault must not stop every converge until jr2 is
+    fixed. The skip is never silent: `up` writes one warning line, which names each model that it did not check. `up`
+    still deletes the leftovers of earlier runs, because their Secrets can hold a key. The flag has no config key,
+    because no flag of `up` has one (CI gives `--yes` as a flag). A model that does not work then fails in a run, as
+    `agent.fault`.
+  - **A limit.** The check pod has labels of its own. It does not get the labels of a Harness pod: the Instance Harness
+    Service selects its pods by label, and would send Turns to the check pod. So a NetworkPolicy that selects Harness
+    pods by label does not select the check pod. When such a policy closes the egress of the Harness pods, the check can
+    pass while a Turn fails. jr2's own egress policy, which closes the cloud metadata addresses
+    ([ADR-0076](0076-an-agents-control-credential-lives-only-where-no-workspace-code-runs.md)), gives the check pod one
+    of its own, so it does not have this limit.
+
+**`jr2 down`** removes the instance from the cluster. It asks for confirmation unless `--yes` is given (for CI). `--all`
+takes the operator too.
+
+## Addressing: cluster is address, namespace is identity, the cluster is the record
+
+- **Cluster = address**: always the current kube context (`--context` to override), never recorded in the repo. The
+  instance repo must stay deployable anywhere.
+- **Namespace = identity**: set in `jr2.config.ts` (default: the instance `name`), `-n` overrides. The same namespace on
+  kind and prod is what makes switching painless.
+- **Convergence is claimed only of observed state.** Every layer `up` rolls out is checked against the pod actually
+  serving — the running image must be the intended one before `up` says `converged`. The staleness labels `up` stamps
+  decide whether to spend a build; they are never evidence that the cluster is correct, because a run that reports
+  success off its own bookkeeping can deploy nothing and still congratulate itself. Tag equality is the whole check, and
+  is only sound because the tag is a content address: on the `kind load` path a pod's `imageID` is containerd's manifest
+  digest under a rewritten repo name, comparable to nothing the host holds.
+- **No local target state.** Whether this cluster hosts the instance is derived from the cluster: `up` finds the
+  instance's labeled objects → converge silently (it's home); finds nothing → confirm first-time setup interactively
+  (`--yes` for CI); finds objects labeled as a _different_ instance → refuse. Run-verbs against a cluster with no
+  instance error with "not deployed here — right context?". This is the guardrail against silently pointing at the wrong
+  cluster, without a state file that can itself go stale.
+- **CLI transport**: run-verbs port-forward the Orchestrator Service via the kube API for the duration of the command
+  and read the Instance token from its Secret — kube RBAC is the real gate. `--url`/`JR2_URL` (+ token env) is the
+  escape hatch for ingress-exposed instances and callers without cluster creds.
+- **A dead ADDRESS and an absent IDENTITY are two faults, and resolution names which.** They share a verb and a context
+  but not a fix: one is the network, the other is `jr2 up`. A cluster that answered and holds no instance Secret is "not
+  deployed here — right context?"; a cluster that could not answer at all is "cannot reach the cluster — context X: …",
+  carrying kubectl's own reason. Merging them — which is the easy shape, since both end in no token — sends a user whose
+  VPN is down to go and check a context that was right all along. They are told apart by EXIT CODE, not by matching on
+  kubectl's English: the Secret read passes `--ignore-not-found`, so an absent Secret exits 0 with empty output and
+  everything else (no route, no credentials, RBAC refusing the read) exits non-zero and is reported.
+- **Resolution is bounded; converge is not.** A run-verb waits **10s** to find out whether the cluster is there — the
+  same bound the Harness client puts on its own dial, so the two seats that cross a network agree on what "too long"
+  means — and then fails with the reason above. Unbounded, it does not fail at all: a cluster whose API server accepts
+  the connection and says nothing (a dropped SYN, a sleeping VPN) makes `jr2 runs` hang with no ceiling, which is the
+  one outcome a CLI must never have. The bound is spent twice over, because `--request-timeout` bounds a single server
+  request while kubectl retries API discovery behind it — measured, a 5s flag bought a 25s command — so the flag makes
+  each attempt give up promptly and a process deadline is what bounds the command. It covers the resolution path only:
+  `jr2 up` legitimately waits on rollouts for minutes and says so with its own `--timeout`.
+
+## Sharing is npm; the instance repo is a deployment assembly
+
+Nothing in `jr2.config.ts` is useful to others _by design_ — it is the boundary where shared code meets your repos and
+your cluster. Reusable work is published as npm packages, and a package exports a **Machine**, nothing beside it: its
+Agents are its own actor slots and its Sandbox Image is a `workspace()` option, so a workflow file imports the Machine
+and the parts ride along (ADR-0049). Nobody clones an instance repo to reuse it — `jr2 init` + a dependency is the path.
+`up` typechecks the instance before it builds anything and refuses on errors (ADR-0050): a wrong Agent slot or a
+`customize()` of a part the Machine does not carry is a compile error at the door, not a mid-run failure.
+`jr2.config.ts` stays committed (the instance repo is the GitOps unit, ADR-0008); everything deployment-varying resolves
+from env.
+
+## Considered options
+
+- **Keep host dev mode as an accelerator** (hot reload, in-process debugging) beside a context-targeted everything else.
+  Rejected: the two-worlds split _is_ the footgun factory, and "same folder runs locally and deployed" (ADR-0008) is
+  only actually delivered when there is one world. Inner-loop cost is absorbed by staleness-checked builds; a `--watch`
+  rebuild-redeploy loop can come later.
+- **Split setup from run** (`jr2 install` + `jr2 up`). Rejected: a two-step runbook reintroduces "you forgot step 1",
+  which `up` would have to detect anyway — at which point it may as well fix it.
+- **A committed or locally-cached deployment target.** Rejected twice: committed addresses break repo reuse; a local
+  cache (`.jr2/target.json`) is derived state that can lie. The cluster itself is the record.
+- **`jr2` creating clusters** (`jr2 cluster up`, or an interactive offer). Rejected: with no baked mounts, any vanilla
+  cluster works; creating one is infrastructure provisioning, like installing docker. The no-context error message says
+  `kind create cluster` and that is the whole story.
+
+## Consequences
+
+- The everyday story is two commands ever: `jr2 init`, `jr2 up`. The full CLI: `init`, `up`, `down`, `gc` (ADR-0039),
+  `run`, `runs`, `status`, `logs`, `send`, `kit push` (ADR-0044), plus the workspace verbs `ls`, `ssh`, `rm` (ADR-0009;
+  decided, not yet in the binary).
+- **Simplification must not surprise**: `up` prompts exactly when meeting a cluster that isn't yet home, and run-verbs
+  print the context they're talking to — the ambient-context magic stays visible.
+- The snapshot store (sqlite) moves onto a PVC in the instance's namespace; `.jr2/` shrinks to scratch.
+- `jr2 build` (build + push + render manifests, no apply — the pure-GitOps CI verb) is planned but deferred; `up` in CI
+  covers the interim.
+- The e2e tier (ADR-0010) loses `jr2 dev` as its per-scenario fixture; the fixture boots the Orchestrator's server
+  entrypoint as a host process instead — the same real server the cluster runs, no cluster, no user-facing verb.
+- The justfile returns to kit-repo development only; no instance operation appears in it.

package/docs/reference/decisions/0020-private-ca-trust-is-a-named-concept.md

@@ -0,0 +1,60 @@
+# Private-CA trust is a named concept, not volume plumbing
+
+Running the jr-parity exercise against a real vLLM (2026-07-19) hit an endpoint that is HTTPS-only behind an ingress
+whose cert chains to a private CA. The host trusts that CA; pods do not — so every jr2-owned egress point fails the same
+way: the Harness's model calls, and `jr2 up`'s own provider preflight (which checks from a one-shot in-cluster pod, the
+check pod, ADR-0019). The tempting fix was generic `volumes`/`volumeMounts` passthrough on `sandbox` — the CRD already
+carries both, and `harness.env`/`envFrom` set the precedent of corev1 shapes passed through verbatim. But trusting a CA
+is not pod-shaped config: it is an _intent_ ("this instance's egress must trust this CA") that jr2 alone can thread
+through every place it owns an outbound TLS connection — including the check pod, which no instance-authored volume can
+ever reach. A passthrough would also make the one-intent story a three-part incantation (hand-managed ConfigMap +
+mount + `NODE_EXTRA_CA_CERTS` via `harness.env`), the runbook shape ADR-0019 exists to kill.
+
+## Decision
+
+- **`harness.caBundle` is the surface**: a path to a PEM bundle, **relative to the instance folder**, committed (CA
+  certs are public data — never a Secret). One file may carry multiple certs; that is what a PEM bundle is.
+- **Only `jr2 up` reads the file, host-side.** `jr2.config.ts` is evaluated in two worlds — by the CLI on the host and
+  by the deployed orchestrator in-cluster — so the config carries a _path_, not file contents: the in-cluster evaluation
+  checks presence and never touches the filesystem. `up` fails loudly on an unreadable path.
+- **`up` materializes the `jr2-ca` ConfigMap** beside the harness ConfigMap; both Harness placements mount it at
+  `/etc/jr2/ca` on the **Harness container** and set `NODE_EXTRA_CA_CERTS=/etc/jr2/ca/ca.crt`. CR-level `volumeMounts`
+  land on the primary container by the operator's contract, so the asymmetry costs no operator change: the User
+  Container (user-owned image — same rule as `harness.env`, ADR-0005/0013) never inherits the trust path.
+- **While a held secret exists, the bundle also reaches the Custodian and the Harness's other tools**
+  ([ADR-0059](0059-a-harness-holds-stand-ins-and-the-custodian-holds-the-keys.md)). `jr2 up` adds three keys to
+  `jr2-ca`: `extra.crt` (`caBundle` + the Instance's held-secret CA), which `NODE_EXTRA_CA_CERTS` names instead;
+  `bundle.crt` (the roots + both), which `SSL_CERT_FILE`, `REQUESTS_CA_BUNDLE` and `GIT_SSL_CAINFO` name, because those
+  replace a tool's store; and `upstream.crt` (the roots + `caBundle`, never the held-secret CA), which the Custodian
+  verifies a bound upstream against — a corporate gateway whose certificate chains to the private CA included. "Roots"
+  are the Mozilla roots of the Node that runs `jr2 up`. The ConfigMap is then applied server-side: two copies of the
+  roots are more than client-side apply's annotation holds. The Custodian's control listener is plain HTTP to the
+  Orchestrator's Service and trusts nothing.
+- **The provider preflight trusts the same bundle the same way** (`NODE_EXTRA_CA_CERTS` on the check pod, from its own
+  copy of `jr2-ca`). A preflight that fails where the Harness would succeed — or passes where it would fail — is a
+  broken promise; sharing the mechanism keeps the two verdicts identical. (Node 24's `fetch` honoring
+  `NODE_EXTRA_CA_CERTS` was verified live against the real endpoint before this landed.)
+
+## Considered options
+
+- **Generic `sandbox.volumes`/`volumeMounts` passthrough.** More powerful and precedent-consistent, but it cannot reach
+  the check pod, splits one intent across three keys, and adds a forever-surface for a need that has appeared exactly
+  once. Deferred until a second real need for arbitrary volumes shows up — the CRD support is already there when it
+  does.
+- **`NODE_TLS_REJECT_UNAUTHORIZED=0` via `harness.env`.** Zero kit change, but it disables verification for _every_ TLS
+  connection the Harness makes (git, MCP, everything), and the preflight still fails — `up` would refuse to converge an
+  instance whose pods would work.
+- **Baking the CA into the Harness image.** Contradicts ADR-0018's stock published image — a per-instance CA means a
+  per-instance image, the exact build ADR-0018 deleted.
+
+## Consequences
+
+- Rotating or adding a CA is a `ca.crt` edit + `jr2 up` + pod restart — the same ConfigMap-update path as Agent
+  definition edits (ADR-0018).
+- `NODE_EXTRA_CA_CERTS` _adds to_ Node's trust store, so public endpoints keep working. With no held secret it reaches
+  Node processes alone — an agent shelling out to `curl`/`git` against the private CA needs the system store, which is
+  the Sandbox Image's business. With a held secret, `bundle.crt` reaches curl, git and Python too; the cost is that it
+  REPLACES their store, so a private CA the Sandbox Image trusts on its own, and `caBundle` does not name, stops being
+  trusted by them in that pod (ADR-0059).
+- The orchestrator pod itself (boot-reconcile `git clone`) does not consume the bundle yet; repos behind a private CA
+  would extend the same ConfigMap to the orchestrator Deployment — a deliberate later step, same concept.

package/docs/reference/decisions/0021-workspace-continuity-is-a-lease-that-answers-back.md

@@ -0,0 +1,101 @@
+# Workspace continuity is watched; the lease asserts
+
+Two Sandboxes were deleted by hand under a live Orchestrator (2026-07-19); `jr2 runs` and the visualizer both kept
+reporting the runs as active, parked on their gates, indefinitely. The gap was structural rather than a bug: the
+Orchestrator's two conversations with the cluster ran in opposite directions and on opposite triggers. The keepalive
+lease (ADR-0001) was **level-triggered and write-only** — a `setInterval` per owned Sandbox, in a process-global map,
+stamping `jr2.dev/keepalive` forever. The restore-reconcile probe (ADR-0012) was **edge-triggered and read-only** — one
+`exists()` per entry into `running`. So the Orchestrator asserted liveness continuously and learned the truth almost
+never, and the state most likely to outlive its Sandbox — a body parked on a Gate for hours — is precisely the one that
+never re-enters `running` to ask again.
+
+Worse, `exists()` asked the wrong question. A run does not depend on a Sandbox CR existing; it depends on **the pod it
+attached to still being that pod**. `work` is an `emptyDir`, so an eviction or a node loss takes the clones, the
+worktrees, and every unpushed commit — while the operator recreates the Pod under the same name, leaving the CR present,
+the Service DNS resolving, and every path in `handles` still valid. Presence answers "yes" to a workspace whose contents
+are gone. That divergence was undetectable at any point in the run's life, restore included — the exact "resume into an
+inconsistent world" ADR-0012 exists to prevent.
+
+The chaos tests (2026-09-27) found the other ways to lose a workspace. A node-pressure eviction leaves the pod object in
+place, `Failed`, with the same UID — no deletion, no replacement, so neither presence nor identity changed, and the run
+hung. A Harness that crash-looped under a live pod only ever refused connections, so a Turn's `wait` reconnected without
+end. And a restarted Harness answered a `continue` with no memory of the conversation and no fault.
+
+## Decision
+
+- **The unit of reconciliation is _continuity_, not existence**: is the workspace I am keeping alive still the one the
+  body attached to? Every way to lose it collapses into the one `workspace.lost` event, and the body's policy decides,
+  unchanged.
+- **A Sandbox has one pod for its life.** The operator creates the pod once. When that pod is gone (deleted, its node
+  lost) or terminal (`Failed` after a node-pressure eviction, `Succeeded` or `Failed` after a node shutdown), the
+  Sandbox goes to the terminal phase **`Lost`**, with the pod's own reason and message (for a deleted pod, the last it
+  saw). It never returns to `Pending` and is never given a second pod: `work` is an `emptyDir`, so a new pod could
+  continue nothing, and it would hold a whole Size for nobody while hiding the loss. The CR stays, so `jr2 status` can
+  say why, until teardown or the idle GC. A Sandbox delete waits for its pod, because a retry can take the name of a pod
+  that is still going away.
+- **The operator judges identity, because it knows which pod it created.** It still publishes `status.podUID`; the
+  Orchestrator reads `Lost`, not a UID comparison.
+- **The watch answers; the lease asserts.** The Orchestrator watches its Sandboxes
+  ([ADR-0063](0063-the-orchestrator-watches-the-cluster.md)). A Sandbox gone or `Lost` is `workspace.lost` within
+  seconds, carrying the reason — or a provision fault, if the Workspace was still `placing` or `provisioning`
+  ([ADR-0064](0064-a-workspace-waits-for-capacity.md)). The lease renewal is a write only — one merge patch every 5
+  minutes, ±20% jitter — the assertion that keeps the operator from reaping (ADR-0001). A dropped watch is unknown,
+  never loss: fabricating loss would settle live runs holding real work the first time the API server hiccuped, so the
+  loop re-lists and reconciles instead.
+- **A Harness restart ends its Turns, not its Workspace and not its conversations.** When the watch sees the Harness
+  container's restart count pass its value at a Turn's admission, every Turn waiting on that Harness fails at once:
+  `memory limit` if the last end was `OOMKilled` (ADR-0061), otherwise
+  `Turn lost (Harness restarted: <reason>, exit <code>)` — unless the Turn's state is busy or its pick's answer is
+  unread, which cuts the Turn and faults nothing (ADR-0031). The pod, `/work` and the conversations' directory survive,
+  so the Workspace is not lost and the Agent's next Turn continues its conversation (ADR-0031); a conversation the
+  rebuilt Harness cannot read gets a `conversation-new` notice (ADR-0062), and a Harness that never comes back faults
+  that admission inside its bounded window, with the kubelet's reason. `wait` keeps reconnecting without a deadline —
+  every real end now has a watch signal, so a timer would only be a guess at how long a slow Turn may take.
+- **The lease is an invoked actor**, one per workspace, owning both halves: it renews, and it subscribes to the watch
+  for its Sandbox. Renewal covers the Sandbox from its write to its teardown, `placing` included, so a Sandbox that
+  waits for capacity is never reaped as abandoned (ADR-0064); Continuity is judged from `running`, once there is an
+  attached pod. Its lifetime is the state's lifetime, which xstate already manages: it re-invokes on snapshot restore,
+  so restore-reconcile stops being a special case and becomes the first tick of the normal loop; and it stops on every
+  exit — body final, run stopped, run faulted.
+
+That last point deletes machinery rather than adding it. Three mechanisms with three different owners and lifetimes —
+the process-global heartbeat map, the one-shot probe, and `release(runId)` (which label-queried the cluster to stop
+timers the in-process map had lost track of, wired into `RunHost`'s error channel) — collapse into one actor. A faulted
+run now stops its lease because it stops its actors; the abandoned pod stays inspectable and ages out of the operator's
+idle timeout on its own. A cancelled run's Workspaces are destroyed by the cancel unless it says `keep` (ADR-0025). The
+behavior `release()` was written to produce is emergent.
+
+## Considered options
+
+- **The renewal answers back** (the first form of this decision): `kubectl annotate --overwrite -o json` returned the
+  patched object, so asserting and learning cost one round trip. Replaced: detection then ran only at the lease interval
+  (an eviction or drain surfaced 4.6–4.7 minutes late in the chaos tests), and it rode a `kubectl` process the
+  Orchestrator no longer starts (ADR-0063).
+- **Keep the probe, just run it on a timer.** Simplest diff: the probe is already a `fromCallback` with teardown, so it
+  is a `.then` → `setInterval`. Rejected because it leaves the heartbeat map, `release()`, and the write/read split all
+  standing, and doubles the API traffic against the same object — the debt, untouched, plus a timer.
+- **The operator recreates a lost pod, and the Orchestrator compares `podUID`** (this ADR's first form). Rejected: a pod
+  that ends in place keeps its UID, so eviction was missed; and the recreated pod was empty, held a Size, and served
+  nothing.
+- **A deadline on `wait`.** Rejected: any number either ends a slow Turn or leaves a dead Harness hanging for that long;
+  the Harness's restart count is the exact signal.
+- **Treat a Harness restart as a lost Workspace.** Rejected: `/work` survives, and a human or the next Turn can use it.
+- **A deletion tombstone** so a client can tell "reaped" from "never existed". Unnecessary once continuity is the
+  question: both answers are `{present: false}`, and the body's policy is the same either way.
+
+## Consequences
+
+- Detection latency for a lost workspace is the watch's: milliseconds (ADR-0063). The lease interval (default 5m, well
+  inside the 30m idle timeout) now bounds only how long an orphan waits to be reaped.
+- A run that faults after its Sandbox was placed stops its lease; the operator reaps the CR one `idleTimeout` after the
+  last renewal. A Sandbox that was never placed is deleted at once (ADR-0064).
+- A Menu-only Agent's Turn on the Instance Harness is outside the watch: its StatefulSet recreates the pod, the new
+  process rebuilds the conversation from its volume, and the cut Submission settles `failed` with the typed
+  `harness_restarted` error ([ADR-0031](0031-menu-only-agents-run-on-the-instance-harness.md)) — the Agent actor reads
+  that as "Turn lost", the same fault the watch gives a Sandbox. An Instance Harness that crash-loops without end still
+  leaves such a Turn waiting.
+- The lease is per-workspace, not per-run: a workflow with concurrent workspaces gets one actor each, and each is lost
+  independently. This falls out of invoking it beside the body rather than owning it at the host.
+- `SandboxPort` has no optional method: `renew` is a write, `continuity` is a subscription to the watch, and
+  `memoryFault` (ADR-0061) answers undefined from a backend that cannot see the container. The `jr2.dev/run` label
+  survives for `jr2 ls` and as the watch's selector, no longer load-bearing for lease bookkeeping.