@jr2/orchestrator 0.3.0 → 0.5.1

package/docs/reference/decisions/0001-operator-owned-generic-sandbox.md

@@ -0,0 +1,113 @@
+# Sandboxes are provisioned by a Kubernetes operator via a generic CRD
+
+A `Sandbox` custom resource describes infrastructure only — a primary container, a generic list of additional sidecar
+container specs, volumes, resources, secrets, idle timeout — and a custom operator reconciles it into one Pod — created
+once, never recreated: a pod that is gone or terminal makes the Sandbox `Lost` (ADR-0021) — plus a Service, reporting a
+`status.endpoint` the Orchestrator uses to reach the Harness. The CRD names the Repos a Sandbox needs by identity —
+enough for the operator to place the pod and mount each node cache read-only (ADR-0051) — and knows nothing about
+clones, worktrees, or Agents; those are layered on by the Orchestrator after the Sandbox reaches `Ready`. The operator
+also publishes what the Orchestrator needs to know about the pod onto the Sandbox's status — `podUID`, a scheduling
+condition with the scheduler's message, the Harness container's restarts and last terminated reason, and the same for
+the Sandbox Effector's container — because the Orchestrator watches Sandboxes and never reads a Pod
+([ADR-0063](0063-the-orchestrator-watches-the-cluster.md)).
+
+We chose the operator over the Orchestrator calling the Kubernetes API directly because the custom resource _is_ the
+durable desired state: Sandboxes survive an Orchestrator restart, and garbage collection / readiness / retries live in
+one reconciler instead of being threaded through the Machine. We kept the CRD generic (rather than coding/worktree-aware
+with an init-container clone) so the operator stays workflow-agnostic — matching jr2's thesis that the framework is a
+grab-bag of composable pieces, with git/worktree being just one piece a coding Machine bolts on. The operator is
+git-aware to exactly the extent of the `Repo` CRD and the cache agent that keeps a bare checkout per node (ADR-0051),
+and of `gitCredentials`, the push Secrets a Sandbox names, which it projects into the Sandbox Effector (the `sidecars`
+entry named `effector`) alone (ADR-0051, ADR-0068): a fetch is infrastructure, shared by every Sandbox on the node; a
+worktree is a Machine's. The cost is an extra runtime step (worktree setup as its own state that can fail independently)
+and an operator to build and maintain.
+
+## The Harness is the primary container; sidecars are opaque fragments
+
+The CR's `spec.image` is the **Harness** — jr2's own server hosting the instance's Agents (ADR-0005/0018/0027).
+Everything else in the pod rides `spec.sidecars`: plain Kubernetes `Container` fragments (image, ports, env,
+volumeMounts) the operator schedules **without understanding them** — exactly as a Deployment's pod template carries
+arbitrary containers without the controller knowing their roles. That is how the Custodian (ADR-0013, ADR-0059) and the
+User Container (ADR-0005) land in the pod with zero operator awareness: `kubectlSandbox` compiles them into the sidecar
+list when it builds the CR. Containers cannot be added to a live pod (native containers, that is), so everything a
+Sandbox will run must be in the spec at spin-up.
+
+One Harness **image** serves many **Agents** — the Harness resolves model, instructions, and its Menu at runtime without
+a rebuild, reading the definition each admission carries (ADR-0018, ADR-0049) and reading the Menu through the Custodian
+per turn (ADR-0013). So the Orchestrator injects a fixed Harness image plus **config** (env + the prepared worktree),
+never a per-Agent image build. The HTTP prompt body itself carries only `{message, images}`, so anything persona-shaping
+is set at provision time, not per request.
+
+## Idle GC: a renewed lease, not a TTL
+
+The operator reaps **abandoned** Sandboxes — ones whose Orchestrator is gone — as the backstop behind the Orchestrator's
+own teardown (ADR-0012). "Abandoned" is defined by a lease: each live workspace renews a keepalive annotation
+(`jr2.dev/keepalive: <timestamp>`) on its CR, and the operator deletes a Sandbox only once `spec.idleTimeout` (default
+`30m`) has elapsed since **max(creation, last keepalive)**. A run parked on a Gate for hours keeps its Sandbox — its
+lease is still renewing — while a `kill -9`'d Orchestrator's Sandboxes reap one idle-timeout later. An Orchestrator that
+restarts within the timeout re-attaches and resumes renewing; one that stays down longer finds the CR gone and delivers
+`workspace.lost` to the restored body. Creation counts as the initial lease, so a CR whose run faults before it ever
+renews is still reaped on schedule. The lease is renewed from the CR's write, so a Sandbox that waits for a node or for
+quota is never taken for abandoned (ADR-0064).
+
+There are no ownerReferences in this scheme — nothing in the cluster represents a run, so liveness has to be asserted,
+not referenced. The lease only asserts; the Orchestrator learns from its watch of the Sandboxes (ADR-0021, ADR-0063).
+
+## The operator scales with Sandboxes, not with the cluster
+
+The operator is deployed once per cluster (ADR-0008), so its cost must follow what jr2 runs, not what the cluster runs.
+The scaling review (R7, 2026-09-30) found it fine at 40 Sandboxes and predicted the breaks at hundreds, or on a busy
+cluster; these are the rules.
+
+- **The operator caches only what it labels.** controller-runtime keeps an in-memory copy of every object a controller
+  watches; the Pod and Service informers take the selector `app.kubernetes.io/managed-by=jr2-operator`, which every
+  object the operator creates carries, so memory and watch traffic are O(Sandboxes) whatever cluster the operator lands
+  in. Sandboxes and Repos are jr2's own kinds and are cached whole. The one read that must not trust the cache — the
+  confirmation before a pod is declared lost (ADR-0021) — goes past it through the API reader.
+- **Reconciles run in parallel, at fixed numbers.** controller-runtime never reconciles one Sandbox twice at once and a
+  reconcile holds no shared state, so the Sandbox controller runs 16 workers, the Repo controller 4, and the client's
+  rate limit is 100 QPS with a burst of 200, so a burst of CRs is placed in a fraction of a second instead of one API
+  round trip at a time. Kit values, not config: nobody has a reading to tune them by, and the API server's own priority
+  and fairness still governs.
+- **A Repo event wakes only the Sandboxes it changes.** A Sandbox reads only its own node's entry of each Repo it names
+  (ADR-0051, ADR-0053), so the Repo watch diffs `status.nodes` and enqueues the Sandboxes naming the Repo whose
+  `status.node` changed; a spec change wakes them all. Without this a cache agent's per-fetch status write woke every
+  Sandbox naming the Repo on every node, S×N per interval and S² on a burst of asks.
+- **The token Secret is owned from birth.** The Orchestrator writes the CR first, then the Secret with the CR as its
+  owner, so no path — an ownerRef patch that fails, an Orchestrator that dies between the two writes — leaves a token
+  behind on the cluster. The pod is created the moment the CR is seen and may reach its mount before the Secret lands;
+  kubelet retries the mount, and the Secret arrives one API round trip after the CR, well before the scheduler and
+  kubelet get there.
+
+Not taken (R15): a parked Workspace releasing its pod. A run parked on a Gate for hours holds a pod for hours; that is
+the problem Agent Substrate exists for, and upstream Kubernetes has no production checkpoint/restore to build it on. The
+one constraint kept: the Sandbox CRD stays close to `agent-sandbox`'s shape, so a backend swap stays possible.
+
+Not taken (R14): a headless Service. A ClusterIP Service programs kube-proxy rules on every node per Sandbox; a headless
+one keeps the name and the endpoint and costs no node anything. Deferred until a cluster with many nodes measures it.
+
+## Known limitations
+
+These are accepted gaps in the current operator, recorded so they aren't silently forgotten. Each is a deliberate
+deferral.
+
+- **Egress is not restricted.** The pod is hardened at the host boundary — `automountServiceAccountToken: false`,
+  pod/container `securityContext` (`runAsNonRoot`, `allowPrivilegeEscalation: false`, drop `ALL`, seccomp
+  `RuntimeDefault`) — and its _ingress_ is the Orchestrator's alone: `jr2 up` converges a NetworkPolicy that admits no
+  other pod to a Sandbox ([ADR-0058](0058-a-harness-answers-the-orchestrator-alone.md)), which kind's `kindnet`
+  enforces. Nothing yet restricts _egress_: an untrusted Agent can still reach the cluster's API server IP (it holds no
+  credential there) and the internet. A default-deny egress policy with an allowlist is the next layer, and ADR-0058
+  records it open. The ADR-0013 boundary does not depend on either: only the token bounds what a pod may _do_;
+  NetworkPolicy bounds where it may _talk_.
+- **Sidecars are plain containers, not native sidecars.** They are scheduled as ordinary `containers`, not Kubernetes
+  ≥1.29 native sidecars (`initContainers` with `restartPolicy: Always`). There is no start-ordering or
+  termination-ordering guarantee between the Harness and its sidecars — the Harness may begin serving before the
+  Custodian is up, or outlive it during shutdown. Start-up is covered without ordering: the Custodian's readiness probe
+  holds the pod's Ready, which holds the Sandbox's, so no Turn is admitted before it serves (ADR-0059). Revisit if
+  ordering becomes load-bearing.
+- **`reconcileStatus` writes status only when it changed.** Every status write is a watch event at the Orchestrator
+  (ADR-0063), so an unconditional `Status().Update` per reconcile would wake it for nothing.
+- **Consumers must gate on `phase: Ready`, not endpoint presence.** `status.endpoint` is populated as soon as the
+  Service exists, before the Harness is reachable; only `phase: Ready` (backed by a readiness probe) means "serving".
+  The `Terminating` phase is best-effort — there is no finalizer, so a fast delete may GC the Pod/Service before the
+  phase is observed.

package/docs/reference/decisions/0002-duplex-streaming-actor.md

@@ -0,0 +1,110 @@
+# The agent actor is a duplex channel; its control plane is Menu tool calls
+
+The `agentRun` actor drives an Agent run as a long-lived duplex channel, not a request/response call: it admits the run
+with `POST /agents/:name/:id`, persists the durable handle host-side, receives the Agent's decisions as events, and can
+feed later turns down to the same conversation. We rejected the simpler `fromPromise`-over-synchronous-`POST` model
+because a synchronous call returns exactly once, at the end — it cannot express human-in-the-loop (mid-run approval,
+steering), and a held hour-long cross-pod connection is lost on any blip or Orchestrator restart.
+
+## Control plane: Menu tool calls, not stream interpretation
+
+The Agent emits the domain events the Machine cares about by **calling tools** — its Menu, derived from the invoking
+state's transitions (ADR-0015), read by the Harness through the pod's Custodian and presented to the model as tools
+(ADR-0013). Events are Agent-authored and semantic rather than inferred from a generic event stream. This was proven
+end-to-end against a real Harness + vLLM (PoC #5): the Agent called its tool, the Machine transitioned, and a gated
+action observably occurred only post-approval.
+
+The split-channel model:
+
+| Direction | Kind                                                      | Mechanism                                |
+| --------- | --------------------------------------------------------- | ---------------------------------------- |
+| up        | control / decision (the workflow's events, ADR-0011)      | Menu tool call → Custodian → delivery    |
+| up        | progress / telemetry (tokens, attempts, "still working")  | Harness stream (lossy, observation-only) |
+| down      | the next turn's prompt (feedback, steering at a boundary) | a new submission on the same iid         |
+| down      | the Turn part, restated (the Allowed picks changed)       | the Harness's read at each step boundary |
+| down      | interrupts (deliberate terminal cancel)                   | `abort` on the wire                      |
+
+Tool calls are Agent-initiated, so the Machine speaks inside a Turn only as an answer. The one exception is the
+**restated Turn part**: when something other than the Agent's own pick changes its Allowed picks — a Gate's answer, an
+outside event — the Harness learns it by reading the surface at each step boundary and appends the new list
+([ADR-0029](0029-a-menu-is-fixed-for-a-conversation-and-a-turn-is-told-its-allowed-picks.md)). It is a pull, not a push:
+nothing new reaches the Harness, and what is restated is fixed text plus Menu names. The residual hard case is
+**unsolicited mid-turn steering** — free text from the Machine to an Agent that has not asked. That remains unsolved,
+and it is the only behavior in that bucket, not a dependency of the whole control plane.
+
+## Durability: the admission is the handle
+
+`send` returns an **admission** — `{ streamUrl, offset, submissionId }` — and that is the durable handle: the host
+persists an `iid → admission` ledger beside the run snapshot in the same save (ADR-0016), never in Machine context.
+`wait(admission)` reconnects from the offset, which is how a restarted Orchestrator re-attaches to an in-flight turn
+instead of restarting it (proven by dropping the actor mid-run and re-attaching; the run completed).
+
+**Stopping the actor is a local abandon, deliberately.** The wire's `abort` (`POST /agents/:name/:id/abort`) is a
+first-class terminal outcome, but actor STOP does not call it: a host shutdown stops every actor, and those runs must
+stay alive server-side for restore to re-attach. Remote abort is reserved for a deliberate terminal act, not a stop side
+effect.
+
+## A held tool result is a poll, never a held socket
+
+A pick that moves the workflow to work inside the Agent's state is answered when the workflow is back at Rest there
+([ADR-0024](0024-an-agents-turn-ends-with-the-state-that-asked-for-it.md)) — seconds or hours later. One request cannot
+carry that wait: the Harness's HTTP client (Node's undici) cuts a request at 300 s with no response headers, and at 300
+s between body chunks, and its error then told the Agent "the pick never reached the Orchestrator — try it again" for a
+pick that WAS delivered (measured: cut at 302 s, both timers, and through the real turn loop at 301 s). An Orchestrator
+restart mid-request is a reset after the request was sent, which the Custodian rightly does not retry and which read as
+"the Custodian could not reach it". A model told either goes round again.
+
+So the answer is **poll-with-progress**. The pick's `POST` answers at once: `outstanding`, with a delivery id. Inside
+the one tool call, the Harness long-polls `GET /agents/:iid/deliveries/:id` (ADR-0013), about 25 s per request — the
+same register as the Harness's own stream long-poll — until the answer comes. Every timer on the path sees short
+requests; a GET is safe to repeat, so a reset, a 503 from a stopping Orchestrator and a park cost a retry, never the
+pick; and an Orchestrator restart is a longer wait in the same Turn, because the hold is saved with the snapshot and the
+restored host answers the same poll. There is no default time bound on the wait: pi awaits a tool's `execute` with no
+timer of its own, and the Runaway guard counts calls, not time
+([ADR-0035](0035-a-runaway-turn-is-ended-rerolled-once-then-a-fault.md)).
+
+The same rule holds for the `POST` itself. The Harness says "never reached" only when it knows that the Orchestrator did
+not take the pick. A pick that leaves the Agent's state ends the Turn. The Orchestrator's abort of that Turn can land
+before the pick's receipt does, and the abort then cuts a request that the Orchestrator already took. Real models showed
+this on 2026-10-22 (Haiku 5.5 and Qwen, the `@model` claim `noticed`). The run moved on the pick, and the model read
+"the pick never reached the Orchestrator (This operation was aborted) — try it again". In its next Turn, the model said
+that the call failed.
+
+So a pick's `POST` has three cases when it gets no receipt. They give five texts: one for the first case, and two for
+each of the others. The Harness returns the second and the third case as the call's answer, not as an error, because an
+error reads as a failed pick and invites a second one:
+
+- **The request never left the Harness.** Then the call fails, and the error is "the pick never reached the Orchestrator
+  (<detail>) — try it again". Two kinds of failure prove this. The first kind is a transport failure in which no
+  connection was made, so no byte of the request was written. These are a refused connection (`ECONNREFUSED`), a failed
+  name lookup (`ENOTFOUND`, `EAI_AGAIN`), undici's connect timeout (`UND_ERR_CONNECT_TIMEOUT`), and an unreachable
+  network or host (`ENETUNREACH`, `EHOSTUNREACH`) from the `connect` call. The second kind is the Custodian's 502 with
+  `x-jr2-custodian: unreachable`, when all of its Envoy response flags say that it did not forward the request: `UF`,
+  `UH`, `DF` or `URX` ([ADR-0059](0059-a-harness-holds-stand-ins-and-the-custodian-holds-the-keys.md)). `URX`, the retry
+  limit, is in this set for one reason only. The `events` route retries only on `reset-before-request,connect-failure`,
+  so its last attempt failed before the request was sent. A test of `custodian.ts` holds that route to those two
+  conditions. The Custodian's text for these is "the Custodian could not reach it (<detail>)", in the place of
+  `<detail>`. After an abort of the call, no failure proves this, because the call was the Turn's and the Turn is over.
+- **The Harness does not know whether the Orchestrator took the pick.** Every other failure can come after the request
+  left: a reset, a socket that the other side closed, a headers timeout, an abort of the call, or a Custodian 502 with
+  another flag. These flags are an upstream connection that closed (`UC`), a stream reset (`UMSDR`), an upstream
+  protocol error (`UPE`), a remote or a local reset (`UR`, `LR`), and a downstream protocol error (`DPE`). A `DPE` means
+  that the Custodian did not forward a whole request, but a body framing error can come after the headers went upstream.
+  The Custodian says `UPE`, `UR` and `LR` as its 502 too, so that Envoy's own 503 for them does not read as a stopping
+  Orchestrator's 503. After an abort of the call, the answer is "It is not known whether this pick reached the workflow:
+  your turn ended while the call to `<pick>` waited for the workflow's answer (<detail>). A pick that the workflow takes
+  can end the turn in this way. Do not call `<pick>` again before you read the next message: it tells you where the
+  workflow is now." Otherwise the Turn goes on, and the answer is "It is not known whether this pick reached the
+  workflow: the call to `<pick>` failed before the workflow answered (<detail>). A second call to `<pick>` can run the
+  pick twice." It gives no order to call again, because a pick with no target runs again when it is sent again. And it
+  does not say how the model can find out: a guard back, a self-transition and an unchanged list of Allowed picks all
+  look the same as a pick that did not arrive.
+- **The Orchestrator took the pick, and its answer was not read.** This is an outstanding pick whose poll an abort of
+  the call cuts, or a 2xx whose body did not arrive. A 2xx is the Orchestrator's own word that it took the pick. The
+  answer is "This pick reached the workflow, and your turn ended before you read its answer. Do not call `<pick>` again
+  before you read the next message: it tells you where the workflow is now." When the call's own abort was not the
+  cause, "your turn ended before you read its answer" is "its answer did not reach you (<detail>)".
+
+Each other answer of the Orchestrator keeps its own text, because it is a receipt or a refusal and not a lost request: a
+503 from a stopping Orchestrator (the pick was not taken — try it again, ADR-0042), a 409 while a pick is outstanding
+(ADR-0024), and a 404 when the Turn is over (ADR-0026).

package/docs/reference/decisions/0004-worktree-provisioning-model.md

@@ -0,0 +1,124 @@
+# A Sandbox's workspace is a private local clone of a read-only cache, not a shared worktree
+
+Each Sandbox gets its **own** `.git` by running a local `git clone --shared` from a canonical bare checkout that is
+mounted **read-only**, rather than a `git worktree add` against a single shared `.git`. The clone borrows objects from
+the checkout (via alternates) for fast, disk-cheap setup but owns its refs, index, HEAD, config, and — critically — its
+**lock files**, so a Sandbox contends only with itself. The checkout it borrows from is a **node-local cache** the
+operator keeps (ADR-0051), one per repository per node, so a Sandbox can be scheduled on any node.
+
+## Why
+
+jr gave every feature a `git worktree`, but all worktrees resolved through **one** `default/.git`. Git serializes access
+to that shared state with lock files, so concurrent commit/fetch/rebase/ref updates across worktrees blocked each other
+_through the filesystem_. That contention is the bottleneck the whole per-pod isolation design exists to kill — so the
+worktree model had to change, not just move into pods. A private `.git` removes the shared lock surface **by
+construction**; that is the load-bearing claim, and it does not depend on a measurement. A PoC corroborated it directly
+(same concurrent churn: lock-retries present with a shared `.git`, zero with private clones) and confirmed setup stays
+worktree-cheap (`--shared` `.git` ~7M vs a worktree's ~7M, vs a full copy ~33M).
+
+The isolation is the point, not the speed: the read-only source plus a private pod-local clone is what confines an
+Agent's blast radius to its own pod (it physically cannot write — or `gc` — the canonical source or any sibling's work),
+which is the Sandbox thesis (GLOSSARY.md) applied to git.
+
+We keep jr's worktree _ergonomics_ — fast spin-up, a working dir that looks like a totally normal checkout, one
+canonical checkout everything branches from — because borrowing objects sets up about as cheaply as a worktree while
+giving an independent `.git`.
+
+## The `--shared` invariant, and how it is enforced
+
+A `--shared` clone reads the cache's object store via `objects/info/alternates`; if an object it borrows is ever deleted
+from the cache, the clone corrupts (a PoC reproduced this by `gc`-ing a writable checkout under a live clone). So the
+one invariant: **objects in a cache are never deleted while clones borrow them.** Deletion only happens via
+`git gc`/`prune` — including the `gc --auto` that porcelain commands (`fetch` among them) run past a loose-object
+threshold, which prunes unreachable objects (e.g. commits a force-push orphaned that a long-lived clone still borrows).
+Enforcement, by side:
+
+- **Sandboxes cannot violate it**: the cache is mounted read-only, and gc is a write. Structural.
+- **The cache agent is the one writer per node, and it is config-pinned**: it sets `gc.auto=0`, `gc.pruneExpire=never`,
+  and `maintenance.auto=false` on every checkout it clones **or adopts**, before any fetch — so neither its own fetches
+  nor a human running git inside the checkout can trigger object deletion. Only a deliberate `git gc --prune=now`
+  defeats it, which is `rm -rf` territory.
+- **Maintenance that deletes no object is allowed, and it is the agent's** (scaling review R13, 2026-09-30). A cache
+  that is only ever fetched accumulates a pack per fetch until every object lookup in every borrowing clone opens
+  thousands of packs. So the agent consolidates on its interval, once loose objects or packs pass a count:
+  `git repack -a -d -k` (one pack holding EVERY object, unreachable ones included — `-k` is the invariant, spelled as a
+  flag: a force-pushed branch's commits stay for the clone that borrows them), `git commit-graph write` and
+  `git pack-refs --all`. Git is built for readers under a repack — an open pack stays readable after its unlink, and a
+  missed object re-scans the pack list — so a borrowing clone never sees an absent object. The rule for anyone who edits
+  that command: no `--prune`, no `gc`, and `gc.pruneExpire=never` stays.
+- **Eviction is deletion of the whole checkout**, never of objects inside one, and only once no pod on the node mounts
+  it (ADR-0051). A live clone never loses what it borrows.
+
+The fallback where borrowing can't be made safe (a source jr2 doesn't control) is a **full copy** — fully independent,
+gc-proof, at full disk and the slowest setup. An immutable generation-swap scheme (refresh = a new snapshot, old
+generations released as their Sandboxes drain) was considered again as OCI image volumes in ADR-0051 and rejected there:
+it buys nothing gc pinning does not, and a repack costs a full re-upload.
+
+## Refresh is fetch-in-place; the `Repo` resource is the catalog
+
+The catalog is the set of `Repo` custom resources in the Instance's namespace (ADR-0051): the Orchestrator creates one
+per repository its Machines bind, at boot from the walk and at first attach for a per-run url, and the operator
+reconciles each onto the nodes that need it. A cache is cloned when missing and fetched when present — on the resource's
+interval, on demand before every attach, so a Workspace starts from the remote's now, and on demand for every fetch run
+inside a pod — the worktrees' `origin` asks the cache, and the cache asks the remote (ADR-0053). `fetch` only **adds**
+objects — safe under the invariant by nature. A repository that fails to sync degrades that repository, never the
+Instance: a cold clone that fails fails the one provision that needed it, pointedly; a fetch that fails on a warm cache
+lets the attach proceed on what the cache holds, announced as stale
+([ADR-0048](0048-the-orchestrator-boots-without-its-repos.md)). The cache agent runs in-cluster, so every repository
+must be **fetchable from the cluster** (an HTTPS token or deploy-key Secret for private ones — ADR-0019, ADR-0047); a
+working copy that exists only on someone's host is not a valid source. Local-path urls are cloned like any other url,
+never used in place. A checkout found in a node's cache directory with no `Repo` resource is adopted and pinned but not
+refreshed, and is evicted like any other once nothing mounts it.
+
+A repository is identified by its url — host plus path, with scheme, user, a trailing `/` and `.git` dropped — so every
+spelling a Machine writes for one repository lands on ONE cache per node, and the expensive clone happens once per
+repository per node. The directory name is derived from that identity, printed beside the url by `jr2 status`, and never
+chosen by a human; there is no name to derive twice or to orphan.
+
+## Storage shape: a cache per node
+
+The contract every Sandbox needs is narrow — _a read-only directory per repository it attaches, on the node it runs on_
+— and the writable side is always pod-local (`emptyDir`), which scales with agent count by construction. The backing is
+a **hostPath directory per node** that the Instance's cache agent (a DaemonSet) populates and the operator mounts
+read-only into Sandboxes, with a soft affinity toward nodes already holding the repositories a Sandbox names. A node
+that lacks one clones it on first need — the image-pull model, and the reason the cluster's node count is not a limit on
+Sandbox placement. A single-node cluster (kind) is the one-node case of the same model; there is no separate volume path
+to keep.
+
+Read-only-ness is load-bearing twice over: nobody can write the shared thing (no write contention), and nobody can `gc`
+it (the structural half of the invariant). Shell-in (`kubectl exec`) needs none of this — the cache exists purely for
+fast setup.
+
+## Where setup runs
+
+Worktree provisioning is a post-`Ready` step owned by the Orchestrator: the Sandbox CRD names repositories by identity
+so the operator can place and mount them, but it knows nothing about clones or worktrees (ADR-0001, ADR-0051).
+`workspace()`'s attach is one idempotent in-pod script the Harness runs on `POST /attach` (ADR-0012,
+[ADR-0063](0063-the-orchestrator-watches-the-cluster.md)), once `Ready` says every cache is present and fetched.
+
+## In-Sandbox layout: worktrees off the per-Sandbox clone (gwtmux convention)
+
+"Not a shared worktree" above means _not a worktree off a fleet-shared `.git`_. **Inside** a Sandbox, worktrees are the
+intended ergonomic — taken off the Sandbox's **own** clone. The per-Sandbox clone _is_ the `default/`, and branch
+working trees are its siblings, matching the layout [gwtmux](https://github.com/snapwich/gwtmux) already expects, under
+the slot the Machine gave the repository (ADR-0051):
+
+```
+/work/<slot>/default/        the per-Sandbox `git clone --shared` (holds the pod-local .git; the default branch checked out)
+/work/<slot>/<branch>/        `git worktree add` siblings, one per branch worked
+```
+
+Because the layout is identical to the one used outside the cluster, existing worktree tooling works **unchanged** when
+you exec into a Sandbox. Object resolution falls through: a branch worktree shares the per-Sandbox `default/.git`, whose
+alternates point at the read-only cache — existing objects are read from the cache, new commits land in the pod-local
+`.git`. `default/` is a checkout of the Repo's default branch, as gwtmux's `default/` is outside the cluster: work
+happens in the branch worktrees, and `default/` is the reference tree a human or a tool finds where it expects one. Not
+`--no-checkout`: an empty working tree reads as every tracked file deleted to `git status`, which misleads an Agent that
+looks. The branch name `default` is refused by the spec guard: the branch Worktree is a sibling of that directory, never
+it. `origin`'s push url is the binding's own spelling, so a Machine that bound over ssh pushes over ssh even when the
+cache was cloned over https.
+
+This does **not** reintroduce jr's contention: the shared `.git` here is _per-Sandbox_, and it is accessed serially
+under the standing invariant that **one Agent runs at a time per Workspace** — structural since ADR-0012 (one sequential
+body per workspace). A Machine that runs concurrent Agents in one Sandbox, or an Agent that spawns parallel git-touching
+subagents, reintroduces contention (pod-scoped, not fleet-wide) and must opt into separate Sandboxes instead.

package/docs/reference/decisions/0005-sandbox-pod-composition.md

@@ -0,0 +1,230 @@
+# The Sandbox pod's containers: Harness, Custodian, an optional User Container, and an optional Sandbox Effector
+
+A Sandbox pod composes up to four containers around one shared worktree volume (`/work`):
+
+- **Harness container** — the CR's primary container: the Sandbox Image (user-built or user-brought,
+  [ADR-0037](0037-an-instance-builds-its-sandbox-images-jr2-injects-the-harness.md)) with jr2's runtime mounted at
+  `/opt/jr2` and the command overridden to start the Harness. The Agent's toolchain lives here because the Working tools
+  (read/write/edit/bash) execute here — the tools the Agent can reach are this image's. It is also where a human
+  `kubectl exec`s to inspect a run with the agent's exact tools, worktrees, and filesystem.
+- **Custodian container** — jr2-composed sidecar, upstream Envoy configured at provision by the Orchestrator
+  ([ADR-0013](0013-the-agent-reaches-its-machine-through-a-container-it-cannot-read.md),
+  [ADR-0059](0059-a-harness-holds-stand-ins-and-the-custodian-holds-the-keys.md)): the pod's credential holder. It holds
+  the Sandbox token and the held secrets its pod names — model API keys above all — as files, and carries the Harness's
+  traffic out: its Menu and the ask to the Orchestrator, and its outbound HTTPS. It exists as a separate container
+  precisely _because_ the Working tools give the Agent code execution in the Harness container — the credentials live
+  where the Agent cannot read them, and the Harness container's env holds only their Stand-ins.
+- **User Container** — opt-in third seat, composed only when the `workspace()` names its image (a `file:` docker context
+  the Machine ships or a registry ref, the same resolution as the Sandbox Image, no default —
+  [ADR-0049](0049-a-machine-carries-its-parts-and-composes-by-invoke.md)). The authoring surface is that string —
+  `user?: string` among `workspace()`'s options, beside `image` — or the object it widens to,
+  `{ image, resources?: { limits: { memory?, cpu? } } }`, whose only other key is the seat's split of the Workspace's
+  Size ([ADR-0060](0060-a-workspace-states-its-size-and-jr2-reserves-all-of-it.md)): a budget is the pod's business, not
+  the process's, so it puts nothing in the container. Env and ports are not forwarded, because every key jr2 forwarded
+  into the process would be a crack in "jr2 puts nothing in it"; a routable port (the follow-up below) is the known
+  candidate for the next key. In the pod it is the container named `user` (`kubectl exec -c user`). It runs its **own
+  entrypoint, untouched**: jr2 injects nothing into its process — no env, no command, no probe — and overrides nothing;
+  it mounts only what the checkouts need to be checkouts (below). The zero-contract seat, which is exactly why it
+  exists. Two jobs no other seat can do:
+  - **Unattended services.** A container's one command belongs to the Harness in the primary seat (ADR-0037), so an
+    image's own services — an sshd for managed access, an IDE server, a metrics agent — need a container whose command
+    jr2 deliberately does not own. A system built _on_ jr2 that hands people access to their Sandboxes automates through
+    this seat.
+  - **Credential visibility.** A session's secrets — a forwarded ssh agent socket above all — live in this container's
+    private mount namespace, where the Agent executes nothing. The human sshs in with agent forwarding, works, and the
+    socket is gone on disconnect; at no point does it share a filesystem with code the Agent runs.
+- **Sandbox Effector** — a jr2-composed container, added at provision only when an Effect below the `workspace()` is
+  placed there — a `push`, an `http` whose body is a Worktree file, or a credentialed `command`
+  ([ADR-0067](0067-an-effect-is-an-actor-slot-and-runs-at-most-once-unless-marked-idempotent.md),
+  [ADR-0068](0068-an-effector-holds-an-effects-secrets-and-signs-its-wire.md)), which `jr2 up` reads off the Machines
+  and declares in the pod's manifest beside its Held one. In the pod it is the container named `effector`. Its image is
+  the Sandbox Image unless the `workspace()` names `effector: { image }` — a `file:` context or a registry ref, resolved
+  like the User Container's — for the tools and the credentials kept out of the Agent's image. Its command is jr2's
+  binary on the runtime volume (`/opt/jr2/bin/effector`); it runs as its image's own `USER` (ADR-0037's 1000 fallback
+  included), hardened like the Harness. The Agent executes nothing in it: it holds the secrets its Effects name and the
+  credentials its pushes spend, which the Agent never reads. With the default image it runs as the Agent's uid, and that
+  is the rule, not a gap: the containers share no mount namespace and no `/proc`, so the barrier is the container and
+  the read-only mount, not the uid. Inside it, a credentialed command's child — the author's argv, often running the
+  Agent's code — shares the Effector's uid, so the Effector confines it with Landlock to the image's system paths, its
+  own scratch (its `TMPDIR` included; never `/tmp`) and the secret files it names, and ends every process it left before
+  its Effect settles (ADR-0068). No Custodian, no `harness.env`, and no env beyond where its mounts are and its pod's
+  name, address and node.
+
+The Harness and User containers mount `/work` read-write, **`/repos` read-only** — the node's Repo cache, one bare
+checkout per Repo the Sandbox names (ADR-0004, ADR-0051) — **and `/opt/jr2` read-only**, the runtime volume — so human
+and agent see identical files. Neither `/repos` nor `/opt/jr2` is a second exception; each is half of the first. The
+worktrees are `--shared` clones whose alternates resolve objects from `/repos/<key>`, so a seat holding `/work` alone
+holds checkouts whose every borrowed object is missing — git in the User Container dies on "unable to normalize
+alternate object path". And their `origin` fetches through a program on `/opt/jr2` (ADR-0053), so a seat without that
+volume holds checkouts whose `git fetch` dies. The Custodian mounts none of the three — the pod's credential holder has
+no business in the working tree.
+
+What each container mounts of the credentials, stated once:
+
+| Container   | Mounts                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
+| ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `harness`   | `jr2-ca` at `/etc/jr2/ca` (the trust bundles). None of the Custodian's volumes.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
+| `custodian` | its values (the Sandbox token, its pod's `jr2-held-secrets-<pod>`, each user `secretKeyRef` its pod names) at `/etc/jr2/custodian/values`; its leaves (`jr2-held-tls`); its pod's config (its own `<sandbox>-custodian`, and `program.lua` from `jr2-held`); `upstream.crt`.                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
+| `user`      | the checkouts' three volumes, and nothing else.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
+| `effector`  | `/work`, `/repos/<key>` and `/opt/jr2` read-only; at `/run/jr2/secrets` (0400) the secrets its Effects name — its pod's `jr2-effector-<pod>` literals, each user `secretKeyRef` by reference — and at `@git/<name>/` each `git.credentials` entry's push Secret the Sandbox CR names (`gitCredentials`), which the operator projects from that entry's Secret by reference into this container alone (`@` is no character of a secret's name, so no secret's file takes that directory); at `/run/jr2/effector` its key (its own `<sandbox>-effector`) and `effector.json`; an emptyDir at `/run/jr2/scratch` as `$HOME`; at `/etc/jr2/effector-ca`, read-only, the instance's caBundle ConfigMap when the instance ships one (ADR-0020). |
+
+One git wall stays the image's own: git's dubious-ownership guard fires in the User Container whenever its uid differs
+from the Harness's (the attach created the trees), and `safe.directory` is honored only from system/global config —
+files jr2 does not own in this seat. Env-form config (`GIT_CONFIG_*` on the sidecar) was considered and rejected: it
+would crack "jr2 puts nothing in it", and it does not even cover the seat's flagship access path — ssh login sessions
+scrub container env — so the image would still owe a line for sshd while carrying jr2's env for exec sessions, two
+mechanisms where one honest one serves. A User Container image whose sessions run git carries its own
+`git config --global --add safe.directory '*'` (or ships it in `/etc/gitconfig`).
+
+The worktrees' remotes encode which hop each seat can make: `git fetch origin` asks the node cache, and the cache asks
+the remote with the cluster's credential (ADR-0053 — the hop the pod can make, from any seat, with no credential of its
+own), while origin's **push url** is the real remote in the Binding's own spelling — the attach sets it from the slot
+the Machine bound (ADR-0051), so a Machine that bound over ssh pushes over ssh even when the cache was cloned over
+https, and nothing else plumbs it. A push from the Harness or the User Container therefore succeeds exactly when a
+caller supplies the credential — a human's forwarded agent in the User Container — and never for the Agent, because no
+container it executes in holds one (the credential-visibility story above, unchanged). An UNATTENDED publish is the
+Machine's `push()` Effect (ADR-0067): it runs in the Sandbox Effector, which holds the push credential of the
+`git.credentials` entry the Repo Slot's url falls under — the provision names that entry's Secret on the Sandbox CR and
+the operator projects it (ADR-0051) — and pushes to that url, never to origin's push url from the Agent's config. A
+`git.credentials` entry's credentials never enter the Orchestrator: `jr2 up` writes the push credential — a
+`push: { token }` or `push: { sshKey }`, or the clone credential where the entry has no `push` field — into that entry's
+own push Secret (`jr2-git-push-<hash of match>`), which only the Effector is given, and the clone token into the clone
+Secret `jr2-git-<hash of match>`, which only the cache agent reads (ADR-0051, ADR-0068). The Orchestrator names both and
+holds neither. Opening a pull request is an `http` Effect the author writes; jr2 has no forge client.
+
+## Sharing `/work` across uids
+
+jr2 sets `runAsUser` nowhere — each image's own `USER` decides its seat's uid — so the two writing seats may disagree,
+and plain POSIX modes would then make the other seat's files read-only. jr2 closes that with two mechanisms it happens
+to own, unconditionally (both are inert when the uids already match):
+
+- **The pod carries `fsGroup` — the work group, default `2000`.** The default is convention; the one override lives on
+  the `WorkspaceSpec` (`workGroup?: number`, beside `image` and `user` — pod composition is the spec's business,
+  ADR-0037), for the image whose session users already hold a gid of their own. Never a config key. Any value is safe
+  for the jr2 seats: Kubernetes grants the fsGroup as a supplemental group to every container process, so the Harness
+  writes `/work` whatever the number, and no image's `/etc/group` needs to know it. fsGroup puts a setgid group on
+  `/work`'s root, and setgid propagates down, so everything either seat creates is group-_owned_ by the work group.
+- **The attach stamps a default ACL on every repo root it creates** — `u::rwx,g::rwx,o::r-x`, set by `work-acl`, a
+  static helper on the runtime volume (part of ADR-0037's published `/opt/jr2` surface, static for the same reason `rg`
+  is: it executes on a libc jr2 does not control). A default ACL is inherited by everything created beneath it, and
+  POSIX ignores the process umask where one is present — so every file either seat writes under a repo tree lands
+  group-writable (664/775), with **zero lines in any image**: no umask in a login shell, no jr2 knowledge in a brought
+  User Container. Ownership without writability was the trap — fsGroup alone leaves the other seat group-_read_. On a
+  filesystem without POSIX ACL support the helper warns and changes nothing, and sharing degrades to the umask fallback
+  below.
+
+Group _membership_ is the one thing an image can still owe. `kubectl exec` sessions and a non-root sshd's logins hold
+the work group automatically — both inherit the container's supplemental groups, where Kubernetes put the fsGroup. A
+**root** sshd is the exception: the logins it setuids rebuild their groups from `/etc/group`, so that image puts its
+session user in the work group itself (`groupadd -g 2000 work && usermod -aG work <user>`) — or the spec points
+`workGroup` at a gid its users already hold.
+
+**The promise is scoped to repo trees.** An ad-hoc path at `/work`'s top level — either seat's `mkdir /work/scratch` —
+carries no default ACL, so files there fall back to the writer's umask. The Harness runs at `umask 002` (itself and
+every tool child) and the attach script opens with the same line, so jr2's own writes are group-writable even there;
+what a User Container session writes outside a repo tree is its own business. A Workspace with no Repo Slot (ADR-0051)
+has no repo tree: its steps share `/work` itself, every Agent and `command` of it in the one Harness container as one
+uid, so they need no ACL; a User Container's writes there have this paragraph's limit, until a user of that seat needs
+more.
+
+## Constraints on the seat
+
+The seat is zero-_contract_, not zero-_physics_. What an image brought here must live with:
+
+- **The entrypoint must block.** The pod's restart policy restarts an exited container, so a `CMD` that finishes
+  crash-loops. Run a long-lived process — an sshd, an IDE server, `sleep infinity` at minimum.
+- **The ports of jr2's seats are taken** — one network namespace: `:8080` (the Harness), the Custodian's
+  `127.0.0.1:8081` (control), `127.0.0.1:15001` (egress) and `:15021` (health), and `:8082` (the Sandbox Effector, in a
+  pod that has one). Everything else is the image's. A process that binds `:8082` while the Sandbox Effector is down
+  gets signed requests it can neither forge nor answer (ADR-0068): the Effect fails closed.
+- **Root is allowed.** The hardened security context (non-root, all capabilities dropped, no escalation) applies to the
+  seats jr2 owns — Harness, Custodian and Sandbox Effector, per container — and deliberately not here: the seat's
+  identity is "what jr2 does not own", and hardening it is an opinion. A root sshd that binds `:22` and setuids sessions
+  down to its login user — the standard shape for managed access — runs unmodified; a platform that wants this seat
+  hardened hardens its own image or its namespace's Pod Security profile. The credential-visibility boundary never
+  depended on this seat being unprivileged, only on the Agent executing nothing in it.
+
+## Why separate containers, not one image
+
+- **Image ownership** — the Sandbox Image is the user's (ADR-0037), the Custodian is a pinned upstream image, and the
+  User Container is whatever its owner wants, contract-free because jr2 puts nothing in it. None bloats or constrains
+  the others.
+- **Independent resource limits** — the Workspace's Size is one ceiling for the pod, and a Machine may split it so the
+  User Container gets a limit of its own (ADR-0060); then a runaway build there can't starve the agent (and vice-versa).
+  Without a split the User Container shares the pod budget.
+- **Lifecycle independence** — a Harness crash restarts only the Harness; an interactive session in the User Container
+  survives, and vice-versa.
+
+## What the split is — and is not — a security boundary
+
+The **pod** is the isolation unit (one per Workspace, the k8s north-star): its containers share a kernel, network
+namespace, node, and the worktree volume, and are one trust domain in every isolation sense. The one property a
+container edge does provide is **credential visibility** — the asymmetry ADR-0013 builds on: the Agent executes code in
+the Harness container and none in the others, so a secret that exists only in another container's mount namespace is out
+of its reach. That is why the Custodian holds the Orchestrator credential and the model keys, and it is what the User
+Container offers a human session — with its edges stated plainly:
+
+- It holds only for what stays in the private filesystem or session. Anything on `/work` or listening on `localhost` is
+  shared with the Agent — the pod has one network namespace. The Custodian's listeners are on `localhost` too; what they
+  hand out is a key's USE toward its bound host, never the key (ADR-0059).
+- A live forwarded agent socket is usable by anything that executes as that session's user in that namespace — so a
+  session holding one should not blindly execute what it finds on `/work` (the worktree is agent-authored; sourcing its
+  `direnv`, running its scripts, or trusting its `justfile` volunteers the boundary away).
+- **It is not port isolation.** The User Container's sshd and the Agent's dev server contend for the same ports.
+- **It is not a seat for parallel authorship.** Two writers on one checkout collide as files, whatever the container
+  layout; parallel work belongs on a second checkout — fetch the branch out
+  (`git fetch "ext::kubectl exec -i <pod> -c harness -- git upload-pack /work/<slot>/default"` needs no credential in
+  the pod at all) and push from where the keys already live.
+
+## Considered options
+
+- **No third seat; start services by hand** (`kubectl exec -c harness -- sshd`). Works for a person, rejected as the
+  mechanism: a service a managed system relies on cannot depend on a human's exec, and it plants the session's
+  credentials in the Agent's own mount namespace.
+- **Ephemeral containers as the attach path.** Right for one-off, creds-free debugging (`kubectl debug --target=harness`
+  with a plain image), rejected as the standing mechanism: they cannot run from pod birth, accumulate per attach, and
+  are spent on exit.
+- **A truly minimal Harness** (agent loop only, tools executed in a sibling container) — would need a cross-container
+  execution transport jr2 does not have. Not blocking: the agent toolchain lives in the Sandbox Image; revisit only if
+  that ever becomes a real cost (ADR-0037 records the same rejection from the image side).
+- **umask discipline as the sharing mechanism** (fsGroup + `umask 002` in every writer). Rejected as what the promise
+  rests on: the Harness's half is one startup line jr2 controls, but the User Container's half is a line every image
+  must remember (`[ -d /work ] && umask 002` in a login shell) — a gotcha, and a silent one: forgetting it costs nothing
+  until the Agent cannot edit a human's file, and the failure reads as the Agent's. It also cuts the other way — a
+  session's deliberate `umask 077` under a repo tree would break the sharing the pod exists for; under a default ACL,
+  `/work`'s repo trees are definitionally shared. The umask survives as defence in depth, not contract.
+- **A root init container stamping the ACL on `/work`'s own root** — complete coverage (no repo-tree scoping), owner
+  rights with every capability dropped, kit code only, the standard volume-permissions idiom. Rejected: it puts a
+  `runAsNonRoot: false` container in _every_ pod, so no jr2 pod could satisfy the Pod Security "restricted" profile —
+  today that is forfeited only by a User Container that chooses root, and that choice belongs to the user, not to the
+  composition. The attach already owns the only place repo roots are born, so the narrower stamp costs one script line.
+- **Unifying uids across seats** (document "make both images run the same `USER`"). Rejected: it makes the uid a
+  cross-image contract — every pairing of Sandbox Image and User Container must agree forever, which is exactly the
+  coupling the zero-contract seat exists to remove.
+
+## Consequences
+
+- The generic `Sandbox` CRD composes all of this as plain container specs
+  ([ADR-0001](0001-operator-owned-generic-sandbox.md) holds — the operator stays agent-agnostic); the Orchestrator's
+  `kubectlSandbox` supplies the images. The pod runs two containers, plus `user` when the spec names one and `effector`
+  when an Effect is placed there.
+- Every container but the Sandbox Effector carries a limit from the Workspace's Size (ADR-0060): the Custodian a fixed
+  kit slice, the User Container its split if stated, the Harness the rest. The Sandbox Effector carries no limit of its
+  own: like a User Container with no split, it shares the pod's Size ceiling — the whole Size, as pod-level resources —
+  with the containers beside it, and what share of the Size it gets is not decided. The Harness container alone mounts a
+  memory-backed `/dev/shm` (`sizeLimit` 25% of its share); the User Container keeps the pod's default.
+- jr2 does not gate readiness on the User Container and never restarts the pod for it; it lives and dies by the pod's
+  own policy.
+- **Operator**: `runAsNonRoot` moves from the pod level to the jr2-owned containers — the Harness, the Custodian and the
+  Sandbox Effector (the Custodian states its own context: non-root, read-only root filesystem, no capabilities); the
+  hardened-by-default rule for sidecar specs exempts the `user` container; the pod gains `fsGroup`
+  (`spec.workGroup ?? 2000`). **Orchestrator**: the attach script stamps the default ACL on each repo root _before_ the
+  clone that fills it — inheritance happens at creation, never retroactively. **Harness image**: vendors the static
+  `work-acl` into `/opt/jr2/bin` (ADR-0037's surface). **Harness**: sets `umask 002` at startup. `shareProcessNamespace`
+  stays off — ever — because the credential-visibility boundary depends on it.
+- The Harness container's `/work` writes being group-writable widens nothing: every process that could abuse group
+  access already runs in the same trust domain, and the work group exists only inside this pod.
+- **Routable access is a follow-up, deliberately untaken**: the operator's Service serves the Harness at `:8080`, and a
+  managed system wanting cluster-routed ssh to the User Container needs a declared port there. `kubectl port-forward`
+  reaches any listener in the pod without one, which covers the human case day one.
+- **GLOSSARY.md**: **User Container** is a glossary term.