@jr2/orchestrator 0.4.1 → 0.5.1

package/docs/reference/decisions/0062-jr2-briefs-the-agent-on-its-seat-never-on-its-task.md

@@ -0,0 +1,212 @@
+# jr2 briefs the Agent on its seat, never on its task
+
+The model saw two sources of text: the Agent's `instructions` (identity, ADR-0018) and the Frame's prompt (the Machine
+state's, ADR-0057). jr2 itself spoke only after a failure — the no-signal nudge — so an Agent had no way to learn facts
+only the mechanism knows. ADR-0057 opens with that failure: a weak model never learned its working directory, wrote
+outside the worktree and still called `finish`. ADR-0060 adds more such facts: `os.cpus()` reports the node, not the
+Sandbox's CPUs. ADR-0061 adds events: a kernel memory kill ends a conversation, and the next Agent inherits a Workspace
+whose processes are gone, with no way to know why.
+
+## Decision
+
+- **jr2 has its own text to the model, the Briefing.** It says a fact only if a correct Agent would act differently
+  knowing it **and** cannot find it out for itself: how it acts on the workflow, the Sandbox's CPUs and Size, the
+  working directory, that a Menu-only Agent has no Working tools, and what happened to the last Turn. Nothing about the
+  task — that is the Frame's — and nothing about style or care — that is identity's. Stand-ins are left out: the only
+  change that knowing could cause is an attempt to go around the Custodian.
+- **Two parts, placed by how often they change.** The standing part is the same on every Turn of a seat; it follows
+  `instructions` in the system prompt, in its own delimited section, and is byte-stable (no timestamps, no counters, no
+  Menu names). The Turn part follows the Frame's prompt in a delimited block: the working directory, the notices, a
+  **pick's briefing** for each Allowed pick that has one (jr2's text on the pick's parameter, which the Orchestrator's
+  surface marks on the pick: the Chart briefing of [ADR-0070](0070-a-machine-lends-its-parts-to-a-chart.md), said only
+  on a Turn that may make that pick), and the Turn's Allowed picks
+  ([ADR-0029](0029-a-menu-is-fixed-for-a-conversation-and-a-turn-is-told-its-allowed-picks.md)). So the Briefing never
+  breaks the provider's prompt cache — the Turn part sits in the newest message, which is uncached anyway, and stays
+  unchanged in the history after.
+- **How the Agent acts on the workflow is one standing statement**, true whether a pick ends the Turn or keeps it open
+  ([ADR-0024](0024-an-agents-turn-ends-with-the-state-that-asked-for-it.md)): "You act on the workflow by picking from
+  your Menu: a call to one of the workflow's tools that the newest "Allowed now" lists. Each pick's answer says if your
+  Turn is over. If it is over, stop and call no other tool. If it is not, continue. If nothing is allowed, stop: the
+  workflow prompts you again when something is." The last sentence is safe because the wake asks a model that stopped
+  with nothing allowed when a pick opens ([ADR-0016](0016-agent-turn-mechanics-are-internal.md)). A line that says a
+  Turn ends with exactly one pick would be false for a pick that keeps the Turn open.
+- **The Turn part is written once per Submission, and restated when its Allowed picks change.** When something other
+  than the Agent's own pick changes them inside the Turn, the Harness appends the Allowed picks again at the next step
+  boundary, in a `<jr2-turn>` block of their own: "The workflow's state changed since you last read the Allowed picks.
+  Allowed now: …" ([ADR-0029](0029-a-menu-is-fixed-for-a-conversation-and-a-turn-is-told-its-allowed-picks.md)). It is
+  appended, never a rewrite of the first, so the cache argument above holds for it too. It carries no notice: notices
+  are for the next Turn in their scope. It carries a pick's briefing when the restatement opens that pick.
+- **The Turn part comes last, because a model obeys what it read last.** Measured on the home-lab vLLM model: with a
+  prompt that argued for a pick the Turn did not allow, the Allowed picks placed ahead of the prompt were obeyed 0/10
+  (thinking off) and 5/10 (thinking on); placed after it, 10/10 both. Wording made no difference. The author's task
+  still reads first, as it would with no jr2.
+- **Always on.** No author can turn it off: every fact in it passes the rule above, so removing one can only make the
+  Agent act worse. An `instructions` that repeats a fact is harmless.
+- **A notice is delivered once**, on the next admission in its scope that takes it, then cleared. That is the next Turn,
+  or, for a guard kill that the Agent of an open Turn was not told, that Turn's nudge or wake (below).
+  - **Conversation scope** ("this conversation is new; earlier context is gone") goes to that Agent's next Turn. So does
+    a pick's answer that never reached its call
+    ([ADR-0024](0024-an-agents-turn-ends-with-the-state-that-asked-for-it.md)): the route left the Agent's state, and
+    the next Turn of the conversation the picking Turn continued hears "the workflow left the state of your last turn
+    while your pick was outstanding", with the answer — or, when something other than the route left the state while the
+    pick's work ran, "your pick did not finish: the workflow left the state that asked for your last turn while it ran,
+    and that stopped it" (`stopped`), with an earlier link's reply when one came. No pick-answer notice says the work
+    ran to its end. A fresh picking Turn has no such conversation, and its answer is no notice.
+  - **Workspace scope** ("the Sandbox's processes were killed at its memory limit during `coder`'s Turn; `/dev/shm` was
+    cleared. Anything that was running in the Sandbox at that time is gone.") goes to the next admission of any Agent in
+    that Workspace that takes it. One kill is one notice, however many Turns saw it. Only a kill read on a stream while
+    its wait is live names the Turn whose Submission ran. Both forms end with the whole Sandbox, because every process
+    in it died, a shared Workspace's too, and in the past tense, because the Agent may hear it after processes started
+    again. A kill read from the pod's list names no Turn and says nothing of when or whose: "the Sandbox's processes
+    were killed at its memory limit (peak X of Y); /dev/shm was cleared. Anything that was running in the Sandbox at
+    that time is gone." The list cannot say whose Turn a kill ended: the kill may land after a state exits, or end an
+    attach, a sync or a command. A kernel kill read from the Sandbox's last end (below) says the same, with the limit
+    and no peak: "the Sandbox's processes were killed at its memory limit of Y; …". A Menu-only Agent outside a
+    Workspace receives conversation notices only.
+  - A notice never replaces a fault. The Machine still receives `agent.fault` and decides (ADR-0016); the notice makes
+    sure the next Agent is not working blind.
+- **The Orchestrator is the only keeper of notices**: pending notices are plain data in the run's persisted state
+  (ADR-0007). It sees kernel kills and fresh conversations itself. A kernel kill (ADR-0061, layer 5) has two paths,
+  under one source, the Workspace and the time the Harness container ended. A Turn that the kill ended faults on it, and
+  the fault raises the notice with the Turn's Agent. A kill while no Turn runs faults nothing, so before each admission
+  into a Workspace the Agent actor also reads the container's last end from the Sandbox watch, which the operator
+  publishes: a last end `OOMKilled` raises the notice, which names no Turn. The ledger's `seen` makes the two paths one
+  notice. An actor whose own watch saw the restart reads the last end only when it re-admits the Turn after the cut.
+  When the Turn faults, the fault's notice names the Turn's Agent. A watch that has not listed yet says nothing, and the
+  next admission reads again: the last end stays until the container ends again. A nudge and a wake take no kernel kill;
+  it stays for the next Turn. The re-admission after a cut takes it, as it takes every notice in scope that is not a
+  guard kill. The Harness reports a guard kill (ADR-0061) on two paths, under one id that the Harness gives the kill:
+  - **On a stream.** While a Submission runs, the kill is an event on that conversation's updates stream. The Agent
+    actor reads a stream while it waits on the Submission. It stops when the Agent's state exits, before its abort
+    reaches the Harness. So a stream does not always reach a reader: a kill in that window is on a stream that no one
+    reads. A stopped actor raises nothing from its stream.
+  - **In the pod's list.** The Harness keeps every kill, with `told`: the conversations whose `bash` call returned the
+    kill's verdict to the model. A conversation goes into it when the answer returns, not when the guard fires: a call
+    cut short (the container's end, a restart) told no one. A model that writes text, or waits, is not told. The Harness
+    keeps the list in a file beside its conversations and answers it on `GET /memory-kills`. A kill while no Submission
+    runs is on this path alone: for example, a dev server killed while the workflow waits at a Gate. Before each
+    admission into a Workspace, the Agent actor reads the list. The actor sets the read's limit of 5 seconds, and it is
+    the only limit. A read that never left the Orchestrator, or a draining Harness's 503, is sent again in that time. A
+    read that fails or times out is none, and the next admission reads again: the list keeps the kill. The Orchestrator
+    logs one line when the reads from a Harness start to fail. It logs no line when a read works again, and it logs the
+    failure again only when a read fails after a read that worked. The Agent's uid can write the file, as it can the
+    conversation records: it can change only what its own Workspace is told. So a `peak` or a `limit` that is not a size
+    in the guard's form, or is long, is left out and logged, on the Harness's read of the file and on the Orchestrator's
+    read of the list and of a stream. The kill is kept, so a real kill is never lost, and its notice says no sizes: "the
+    Sandbox's processes were killed at its memory limit; /dev/shm was cleared. …". Only a kill with no id or no time, or
+    one outside the bounds below, is dropped. A stream event from an older Harness has no id; its source is then its
+    place on the stream, exactly as the Orchestrator raised it before.
+
+  Both paths raise the notice with the same source, the Workspace and the kill's id. The ledger keeps every source it
+  raised (`seen`) in the run's persisted state. So a kill read on both paths, or on two streams, is one notice. Each
+  read of the list raises one notice at most: the newest kill's, unless it is raised already. The read marks the older
+  kills in it seen, and they give no notice, so a burst of kills in one read tells the Agent one thing. Kills read on a
+  stream are not folded: each one with different sizes gives its own notice. Two notices with the same words keep two
+  entries in the ledger, and an admission that takes both says them once. The run is saved when a raise changes the
+  ledger, and only then. A read removes nothing, so a crash between the read and the save that keeps it loses no kill. A
+  restart of the Orchestrator raises no kill twice. A restart of the Harness container keeps the file, which is on the
+  pod's emptyDir with its conversations (ADR-0031). A lost pod takes the file with it, but then the Workspace is lost
+  (ADR-0021). The list keeps the newest 16 kills, each with at most 32 conversations in `told`, and the Orchestrator
+  reads at most 256 KiB of it. The Agent's uid can write the file, so the two sides hold the same bounds: an `id` and an
+  `at` of at most 64, an `agent` and an `instanceId` of at most 128, and an `at` that parses as a time. A kill outside
+  them is dropped, and a `told` entry outside them is left out. The Harness counts the bytes of the JSON text, and the
+  Orchestrator counts characters, which are never more, so the Harness keeps no kill that the Orchestrator drops. The
+  Harness reads at most the last 256 KiB of the file. With these bounds the largest list is 149,259 bytes. When more
+  come between two admissions, the oldest is lost, and a newer kill tells the same thing. When more than 16 come in one
+  Turn, the Turn's mark leaves the list too, and every kill in the list then counts as after the mark; a nudge or a wake
+  of that Turn may then carry a kill from before it. The Instance Harness keeps no kill, because it holds no Workspace.
+
+  A notice counts as delivered when its admission is ledgered, so a restart before that point delivers it again and a
+  restart after it does not. Delivered is not yet heard: if the Orchestrator ends the Turn before the Harness reads its
+  surface, the Harness never prompts it (ADR-0026), so the Orchestrator raises that admission's notices again. This
+  holds for a nudge and a wake too: each one must have its surface read again before its notices count as heard. An
+  admission that fails leaves the earlier admission's notices heard. A nudge and a wake re-prompt the Turn within
+  itself, so they take no notice of the kind a Turn takes. They take only the guard kills of the list that come after
+  the newest kill whose `told` names their conversation. That `bash` answer told the Agent that all its processes died,
+  so it covers every kill up to it. They also take no kill from before the Turn started. Such a kill would tell the
+  Agent that processes it started in this Turn died. The Turn's first read of the list sets its mark: the id of the
+  newest kill it saw, or "none". The mark is ledgered with every admission of the Turn, and a restore reads it back, so
+  a nudge or a wake takes only kills after it. Only when the first read failed is the Turn's start the mark, ledgered
+  the same way, with one second allowed for a pod clock that runs behind. A read that failed is "could not tell", never
+  an empty list, so it sets no mark. The kills they do not take stay pending for the next Turn. The re-admission after a
+  Harness restart cut the Turn continues the same Turn, so it takes every notice in scope except the guard kills that
+  this rule leaves out. When its read could not tell, it takes no guard kill at all, as a nudge then takes none: it
+  cannot know which kills its own `bash` reported. The notices that the Turn's admissions carried since the Harness last
+  read the Turn's surface are pending again first, each with the source it was raised with, and this re-admission
+  carries them whatever else its scope leaves out. A notice with no source merges only into a pending one with no
+  source, so an equal notice of another kill never takes its place. When such an unread admission carried the Turn's
+  task, the task follows the re-admit text. When the rebuilt Harness lost the conversation, nothing of the Turn is left:
+  every notice that the Turn's admissions carried is pending again whatever the surface reads said, the re-admission
+  takes every notice in scope, as a Turn's admission does, and the Turn's task follows the re-admit text. It says once
+  that the conversation is new. A restore drops the prompt from a Turn's input, so the task rides the ledgered admission
+  too; a lost conversation with no task to send faults the Turn instead. Whether the Harness still holds the
+  conversation is asked again for up to 30 seconds while the new container does not listen yet or drains, and only then
+  is the answer taken as "held". The continue path asks the same way, so a continued Turn after a restart can wait up to
+  30 seconds before its admission. A Harness in CrashLoopBackOff can take longer than that: the answer is then "held",
+  and it can be wrong. So every admission to a conversation that the Harness was taken to keep says so with
+  `kept: true`: the re-admission after a cut, a continued Turn, and a nudge or a wake. A Harness that keeps no such
+  conversation refuses it with 404 and `kept: false` in the body, and seats none. The Orchestrator takes only that 404
+  as the refusal; another 404 is a failed admission. Without the refusal the Harness would seat a new conversation, and
+  a model on an empty context would read "continue your task" with no task. On the refusal the Turn takes the path of a
+  lost conversation. The re-admission is admitted as a Turn, with the task after the re-admit text and the notice that
+  the conversation is new, or faults when it has no task to send. A continued Turn is told that its conversation is new,
+  and is admitted again. A nudge or a wake is admitted again as a Turn: its prompt is the task alone, because nothing
+  was cut, with the notice that the conversation is new and every notice in scope. Every notice that the Turn's
+  admissions carried is pending again first, as for a lost conversation after a cut. With no task to send, the Turn
+  faults. The Instance Harness's restart is on no watch, so this refusal is the only way a nudge or a wake learns of it.
+  What the Turn's admissions carried, and whether the Harness read their surface, is kept in the Orchestrator's memory
+  alone: after a restore of the Orchestrator before a surface read, nothing is raised again. A fresh Turn's "this
+  conversation is new", which rides an admission alone, is carried again by the next re-admission when the Harness has
+  not read the one before.
+
+  A continued Turn of the same conversation is told, as a notice, a kill that its own `bash` call answered in an earlier
+  Turn. The Agent reads that kill twice, which is true and costs little. The `told` rule could later extend to the first
+  admission of a continued Turn too; it does not now, because a fresh Turn's conversation holds no earlier answer, and
+  the rule would then need the conversation in the ledger's scope.
+
+- **The Harness writes every word.** The admit body carries typed notices
+  (`notices: [{ kind: "memory-limit", scope, agent?, limit?, peak? } | { kind: "conversation-new", reason } | { kind: "pick-answer", pick, cut, answer?, stopped? } …]`,
+  where `peak` comes only with `limit`), and the Harness renders both parts. An outstanding pick's answer is data too —
+  the last reply since the pick, a party's event with its payload or a child's output (ADR-0024) — and its words ("Its
+  answer: `approve` from Gate `review`, with {…}", "No reply came back for it.") are the Harness's, in the receipt and
+  in the notice alike; "it moved the workflow, and nothing ran" is a guard straight back's receipt alone. The standing
+  part needs no wire: the Harness already holds its limits (cgroup and Downward API env), the definition's `workspace`
+  access, and the Menu. The wording lives in one place, and Harness conformance checks it (ADR-0027).
+
+## Considered options
+
+- **Put the facts in the Frame.** Rejected: the Frame is the Machine state's (ADR-0057), and a kit that writes into it
+  makes the author's prompt say things the author did not write.
+- **Leave it to authors' `instructions`.** Rejected: a definition is written before any run, so it cannot know a Size a
+  composer retuned, a working directory, or what happened to the last Turn — and packaged Agents would each word it
+  differently or not at all.
+- **An author switch** (`agent({ briefing: false })`, or notices only). Rejected: under the rule, each fact is one a
+  correct Agent needs; the switch only lets the ADR-0057 failure back in.
+- **Everything in the system prompt.** Rejected: per-Turn content there breaks the prompt cache on every Turn.
+- **Finished text from the Orchestrator.** Rejected: wording split across two processes, and untestable where the turn
+  loop is tested.
+- **Report a guard kill on the stream alone.** Rejected: a kill while no Submission runs, or after the wait stopped,
+  reaches no reader. A dev server killed while the workflow waits at a Gate is then never told to the next Turn, which
+  works blind.
+- **Put that kill on every conversation's stream.** Rejected: the next Admission reads its conversation's stream from
+  its own offset, which is after the event.
+- **Keep in the list only the kills that no stream carried.** Rejected: a kill on a stream whose wait stopped is then in
+  no place that is read. And two streams that carry one kill would give two notices.
+- **The Harness pushes the kill to the Orchestrator.** Rejected: the Harness container is the Agent's, and ADR-0013
+  closes a route from it to the Orchestrator. A push also needs a retry and an acknowledgement, and an idempotent read
+  does not.
+- **Read the kill on the Sandbox watch.** Rejected: the operator publishes what the kubelet says about the container,
+  and a guard kill does not end the container. The Harness has no Kubernetes credentials to publish it.
+- **The Harness words the kill into the next Turn itself.** Rejected: a second keeper of notices, outside the run's
+  persisted state, which cannot know when an admission is ledgered or heard.
+
+## Consequences
+
+- ADR-0060's CPU fact and ADR-0061's memory-limit notice are the Briefing's first users.
+- Each admission into a Workspace costs one more request to its Harness, `GET /memory-kills`. The first read from a
+  Harness that fails is said in one line of the Orchestrator's log.
+- The no-signal nudge stays a re-prompt within a Turn (ADR-0016); it is not a notice.
+- The Briefing is how a Turn learns its Allowed picks: the tools block is the whole Menu, fixed for a conversation so it
+  does not break the cache
+  ([ADR-0029](0029-a-menu-is-fixed-for-a-conversation-and-a-turn-is-told-its-allowed-picks.md)).
+- GLOSSARY.md gains **Briefing**.

package/docs/reference/decisions/0063-the-orchestrator-watches-the-cluster.md

@@ -0,0 +1,83 @@
+# The Orchestrator watches the cluster
+
+The Orchestrator talked to Kubernetes by starting a `kubectl` process for every call (`sandbox-kubectl.ts`). A short
+process can only ask "what is the state now?", so waiting became polling: `get sandbox` every second while a Workspace
+provisioned (and `get pod` every fifth time), `get sandbox` every second while a fetch ask waited (ADR-0053), and
+detection of a lost Workspace only on the Lease's 5-minute renewal (ADR-0021 chose that and named the watch as the thing
+to revisit "if something needs sub-minute loss detection"). The attach was a `kubectl exec` stream per Workspace through
+the API server. The scaling review (2026-09-27) measured 36 concurrent `kubectl` processes and about one core of
+Orchestrator CPU at 40 Workspaces starting, and `workspace.lost` arriving 4.6–4.7 minutes after an eviction or drain.
+Decisions since then need pod facts at once: the scheduler's reason for a Pending pod, a lost pod's reason, and
+`OOMKilled` to name a memory fault (ADR-0060, ADR-0061).
+
+## Decision
+
+- **One in-process client on Node's built-in `fetch`, with no new dependency.** The Orchestrator is always in-cluster
+  (ADR-0019). The Deployment sets `NODE_EXTRA_CA_CERTS` to the ServiceAccount's `ca.crt` (concatenated with the ADR-0020
+  bundle if the Orchestrator ever needs both). The client does plain REST for jr2's own kinds — Sandbox, Repo, Secret —
+  including server-side apply, and one watch loop. Its rules, each observed on a real cluster (2026-09-29):
+  - The token is read from its file on every request, never cached: the kubelet replaced a 10-minute token only about 50
+    seconds before it expired. A `401` re-reads the file and retries once.
+  - A watch asks for bookmarks and `timeoutSeconds=240`. Built-in `fetch` aborts a response body that sends nothing for
+    300 seconds (`UND_ERR_BODY_TIMEOUT`, at 300.8s), and that timeout cannot be changed without undici's dispatcher; the
+    server closing first, and bookmarks about every 60 seconds, keep a healthy watch clear of it. The abort itself is a
+    resume, not a failure.
+  - Every bookmark's `resourceVersion` is kept, so a resume after a quiet period starts from a recent point.
+  - A resume that is too old arrives as an `ERROR` event with code 410 **inside an HTTP 200**. Any `ERROR` event
+    re-lists.
+  - Concurrent writes are capped (8–16). 200 parallel writes each opened a TLS connection and cost about 130% CPU.
+- **The Orchestrator watches Sandboxes only**, one watch on the Instance's namespace — the namespace is the Instance's
+  identity (ADR-0019), and `jr2 up` refuses one another Instance owns — selected by the `jr2.dev/run` label every
+  Sandbox it provisions carries. Observed: event latency p50 11ms, p95 92ms, max 186ms; 0 events lost across a
+  200-object burst and a CRD run; about 75 MiB and 3% CPU for the watcher. **It never reads a Pod.** The operator owns
+  the pod (ADR-0001) and already reads it on every reconcile, so it publishes what the Orchestrator needs onto the
+  Sandbox's status: a scheduling condition with the scheduler's message — or a quota's refusal of the pod create
+  ([ADR-0064](0064-a-workspace-waits-for-capacity.md)) — the Harness container's restarts and last terminated reason,
+  and `podUID`.
+- **The watch drives everything that polled.**
+  - Placing and provisioning wait on watch events: an Unschedulable pod or a quota refusal is a wait with its reason
+    (ADR-0064), a crash-looping Harness is seen as it happens — and named: the Harness container's termination message
+    falls back to its last log lines (`terminationMessagePolicy: FallbackToLogsOnError`), so the fault that ends the
+    provision carries the Harness's own last words, and the Harness says them plainly at startup (a port another
+    container in the pod took: `EADDRINUSE` names the port and the cause). No reason is invented for what a log line
+    already says. And readiness proves the Harness, not the port: the probe is a GET of the Harness's own `/healthz`,
+    because a pod has one network namespace and a socket probe passes for whatever listens there — the nginx that took
+    `:8080` made the pod Ready and the attach met its 404 page (scaling review R16, 2026-09-30). A pod with a Sandbox
+    Effector is Ready only when that Effector serves too: its container is probed on its own `/healthz` (ADR-0068).
+  - Continuity: a gone or `Lost` Sandbox is `workspace.lost` within seconds, and a Harness restart ends the Turns
+    waiting on it (ADR-0021). A dropped watch is unknown, never loss; the loop re-lists and reconciles.
+  - A fetch ask (ADR-0053) is answered by the watch event that carries its landing.
+  - Restore (ADR-0012): the first list after start is the reconcile.
+- **The Lease stays a write.** It is still the assertion that lets the operator reap an orphaned Sandbox (ADR-0001): one
+  merge patch per Workspace every 5 minutes, ±20% jitter so Leases never renew in lockstep after a restart. It no longer
+  carries detection.
+- **The attach is a Harness call, not an exec.** The Harness runs the attach — ACL stamp, `git clone --shared` from the
+  node cache, remotes, `git worktree add` — on `POST /attach { slots, branch }`, authenticated like every Harness call
+  (ADR-0058). No stream per Workspace crosses the API server, and **`pods/exec` leaves the Orchestrator's RBAC**. It
+  also binds work late: a pod already running receives its Repos and branch, the shape standby Sandboxes and Agent
+  Substrate need.
+
+## Considered options
+
+- **`@kubernetes/client-node`.** Informers built in, but a large dependency tree and generated models for every kind jr2
+  never touches, around kubeconfig handling in-cluster code does not need. A contained swap if jr2 ever needs exec or
+  many more kinds: every call stays behind the Sandbox and Repo ports.
+- **`node:https` with its own agent.** Trust scoped to this client alone, but transport code built-in `fetch` makes
+  unnecessary; the Orchestrator sets `NODE_EXTRA_CA_CERTS` for nothing else.
+- **Keep `kubectl`, add `kubectl get -w`.** Fixes detection, not the process per write, and adds a long-lived child per
+  watch.
+- **Watch Pods as well.** Two watches, and the Orchestrator interpreting pod internals the operator already reads.
+- **A jr2 WebSocket API.** Nothing to add: a Kubernetes watch is already a pushed stream of events.
+- **Keep the attach as exec through the new client.** The exec protocol is the hardest part of any Kubernetes client, it
+  keeps one stream per Workspace through the API server, and it keeps `pods/exec`.
+- **Attach from an init container.** Declarative, but it fixes the Repos and branch at pod creation, which rules out
+  late binding, and it must wait on a cache that exists only after scheduling.
+
+## Consequences
+
+- ADR-0021 is rewritten in place: the watch answers, the Lease only asserts. ADR-0012, ADR-0004 and ADR-0053 now read
+  the watch and the Harness attach. ADR-0001: the operator publishes pod facts on the Sandbox's status, and writes
+  status only on change — every write is now a watch event at the Orchestrator.
+- The Harness gains a route that is not about a conversation. It is jr2's own work in the container ADR-0005 gives it.
+- The CLI keeps `kubectl` on the user's machine (`jr2 up`, `jr2 ssh`, ADR-0009); only the Orchestrator stops using it.
+- Tests fake the client with a small HTTP server instead of a `kubectl` stub.

package/docs/reference/decisions/0064-a-workspace-waits-for-capacity.md

@@ -0,0 +1,79 @@
+# A Workspace waits for capacity
+
+Since ADR-0060 a Sandbox reserves its whole Size, so a full cluster shows as a Pending pod, not as eviction. But the
+provision's 120-second budget started at the CR's write, so time spent Unschedulable used it up: the run errored, and
+the CR, pod and Secret stayed until the operator's idle GC. A burst of 10 Workspaces at 500m each on the home cluster
+(2026-09-29) placed 2; the other 8 errored at 120s and left Pending pods behind, one of which could take a node later
+for nobody. Worse, the Lease was renewed only in `running`, so the operator counted creation as the Lease and would reap
+a Sandbox that waited past its 30-minute idle timeout. The scaling review asked two more questions: should jr2 cap live
+Workspaces for the Instance (R4), and concurrent Turns per model provider (R5)?
+
+## Decision
+
+- **A Sandbox with no node waits, and jr2 sets no deadline on the wait.** jr2 cannot tell "full now" from "never fits":
+  the cluster autoscaler, a finished neighbour or a drain can each make room, and the node set moves (ADR-0052).
+  `jr2 up` already warns when no node could hold a Size (ADR-0060). A Machine that wants a bound puts `after` on the
+  state that holds the Workspace; stopping it is the body's policy, as for `workspace.lost` (ADR-0021).
+- **A quota refusal is the same wait.** A ResourceQuota (or Kueue) is the cluster owner's tool; jr2 neither writes one
+  nor replaces it. The operator publishes the refusal on the Sandbox's status and keeps retrying the pod create.
+- **The Workspace Machine has a state for the wait: `placing`**, before `provisioning`. `placing` writes the CR and
+  waits for the pod to be scheduled, with no deadline. `provisioning` waits for `Ready` under the 120-second budget,
+  which now measures only what jr2 controls: the image pull, the Harness start and the Repos. A restart re-enters
+  `placing`: the apply is idempotent and the watch's first list answers at once (ADR-0063).
+- **The wait is visible; its reason is behind the token.**
+  - The state is structure, so the Console lights `placing` and `observe()` carries it with no new field.
+  - The authenticated `RunStatus` gains `waiting: [{ child, on: "node" | "quota", message, since }]`. `message` is the
+    scheduler's or the quota's own words, which can name nodes and taints. `jr2 status <run>` prints it.
+  - The feed has two lines per wait: when it starts, with the reason, and when it ends ("placed after 3m12s"). The
+    duration of a wait this Orchestrator process opened is read on the monotonic clock, so it is never negative. A wait
+    opened before an Orchestrator restart has only its stored wall-clock start, so its duration is two wall-clock reads,
+    clamped at 0. A changed reason updates `waiting`; it adds no feed line.
+  - `jr2 runs` marks a run that has a Workspace in `placing`.
+  - The Machine receives no event: scheduling detail is not the body's concern, and `after` covers a bound.
+- **The Lease covers the Sandbox from its write to its teardown**, in every state, so the idle GC reaps only what the
+  Orchestrator has abandoned (ADR-0001).
+- **Delete what never ran; keep what ran.** A Sandbox whose wait ends without a node — a parent `after`, a stopped run,
+  a faulted run — is deleted at once: nothing on it can be inspected, and left alone it could take a node later. The
+  delete does not touch a name that a live Workspace of the run holds then: a parent that re-enters the state, or
+  invokes the same Machine again, gives its new Workspace the same invoke id and so the same name, and that Workspace
+  has placed it again ([ADR-0073](0073-a-machine-fails-a-run-with-fail-and-an-uncaught-failure-ends-it-failed.md)). A
+  Sandbox that was placed and then faulted keeps ADR-0021's rule: it stays inspectable (logs, `jr2 ssh`) and ages out at
+  the idle timeout.
+- **jr2 caps nothing of its own.** Not live Workspaces (R4), not concurrent Turns per provider (R5). See the options.
+- **A provider's refusal is named.** A Turn that ends because the Harness's provider retries ran out has a fault reason
+  with the fixed prefix `provider limit`, as a memory kill has `memory limit` (ADR-0061), so `jr2 status` and the
+  Machine can tell provider pressure from a bug.
+
+## Considered options
+
+- **Keep a deadline on the wait, or make it a config value.** Rejected: any number either faults a Workspace the
+  autoscaler was about to place or waits for one that never fits, and the Machine's own `after` already states the
+  body's bound where the body can act on it.
+- **An Instance-wide cap on live Workspaces (`sandbox.max`)**, first come first served, persisted: Agent Substrate's
+  request parking in jr2's terms. Rejected: a count of Workspaces of mixed Sizes is not a measure of resources. A spend
+  ceiling is a ResourceQuota on `requests.cpu` and `requests.memory`, exact in resource units and refused before any pod
+  exists, so the autoscaler never scales for it. Placement order among Pending pods of one priority is not promised, and
+  nothing in jr2 needs strict order. Substrate's parking is driven by capacity, not a count.
+- **An Instance-wide cap on concurrent Turns per provider (R5).** Rejected:
+  - A Turn is the wrong unit: it is many model calls with tool runs between them, and a slot held while `tsc` runs
+    blocks the provider while it has nothing to do.
+  - A fixed number is the wrong measure: hosted providers limit tokens and requests per minute, vLLM by batch and KV
+    cache.
+  - The wait already happens in the right place: vLLM queues in arrival order, and the Harness retries twice, through
+    pi-ai, with backoff, and honors `retry-after` up to 60 s (pi-ai 1.4.0). pi-ai retries what the SDKs retry: a 408, a
+    409, a 429, each 5xx, and a connection error (an error with no status). So a quota 429 is asked again too; its
+    settlement keeps the quota's own words, never `provider limit`. The scaling review measured slow Turns at 40
+    concurrent (about 90s against 14s alone), which is throughput a cap cannot add, and no failed ones.
+  - A slow provider cannot make a Runaway: ADR-0035 counts steps and repeats, not time.
+  - If a hosted provider's 429 storm is ever measured, the answer is shared backoff at the Custodian (Envoy's global
+    rate limit), not a Turn count.
+- **Watch for a Pending pod from the Orchestrator.** Rejected by ADR-0063: the operator reads the pod and publishes the
+  scheduling condition.
+
+## Consequences
+
+- ADR-0012's lifecycle gains `placing`. ADR-0021: the Lease is no longer only in `running`, so "attach must stay well
+  inside the idle timeout" goes away. ADR-0001: the operator publishes a quota refusal like an Unschedulable pod.
+- A Workspace parked in `placing` holds no node, only a CR and a Secret. A Workspace that waits a long time still holds
+  its run's place in any `pool()`: the Pool's cap is the Machine's own concurrency policy.
+- GLOSSARY.md gains **Placing**, and **Lease** covers the Sandbox's whole life.

package/docs/reference/decisions/0065-a-persist-survives-a-crash-and-history-is-a-week.md

@@ -0,0 +1,65 @@
+# A persist survives a crash, and history is a week
+
+Every transition of a run persists its whole snapshot (ADR-0007). The scaling review (R6) read the store and listed its
+costs: `JSON.stringify` of the whole run, a synchronous upsert with an fsync on the event loop, a Pool's run as one blob
+holding every worker, terminal rows never deleted, and `restore()` parsing every row before the server listens. It also
+found the words "Postgres opt-in" in GLOSSARY.md, ADR-0008 and ADR-0009, with no such store in the code. Measured on the
+home cluster's chaos Instance (2026-09-30, `local-path` volume): a persist costs **4.5 ms, whatever the blob's size** (1
+KB or 100 KB), because the cost is the fsync, not the JSON; with the write-ahead log and `synchronous=NORMAL` the same
+upserts cost 0.04 ms (1 KB) and 0.18 ms (100 KB). A run's blob is about 1 KB; a Pool of 40 workers is about 100 KB.
+
+## Decision
+
+- **A persist survives a process crash, not a power loss.** The store runs sqlite in WAL mode with `synchronous=NORMAL`:
+  a write is durable once the OS holds it, and sqlite fsyncs only at a checkpoint. A node's power loss can lose the
+  transitions since the last checkpoint; the file stays consistent, and `restore()` treats the older snapshot as it
+  treats any restart — reconcile against the live world, re-attach, or `lost`. The window in which a crash re-POSTs a
+  prompt (ADR-0007's at-most-once sliver) widens from one microtask to one checkpoint, and only on a power loss, which
+  also ends every Sandbox on that node.
+- **The write stays synchronous and on the event loop.** At 0.2 ms for a Pool's blob there is nothing to move off the
+  loop, and the ordering matters: the row is on disk before the Machine's next side effect runs, which is what lets an
+  epoch bump (ADR-0057), an admission (ADR-0016) or an Effect's intent row (ADR-0067) be saved "at the write" — the
+  intent row before its call leaves. A worker thread, an asynchronous driver or a network store would land the row after
+  the next `invoke` fires, on every persist.
+- **One blob per run stays.** The snapshot is a tree that xstate restores from its root, and one row keeps it atomic —
+  with the ledgers beside it (admissions, epochs, notices, timers, outstanding picks, Effects); a Pool's cost at 40
+  workers is 0.2 ms, so its workers do not become rows of their own. The Effect ledger keeps a row for every Effect the
+  run started (ADR-0067), so a Machine that loops through an Effect grows its blob by one row per pass: id, kind and
+  outcome. The output's tail is dropped once the save that carries the Effect's transition is on disk; a push row keeps
+  its published commit, its source, its `squashed` and its sync's answer whole — every later push of its Repo Slot sends
+  them as one of its pairs — and its bundle only while it is its Repo Slot's newest push and its sync did not finish —
+  one bundle per Repo Slot at most. An outstanding pick's hold is the one place tails accumulate: its `chain` keeps
+  every reply since the pick whole until the pick is answered, then it is dropped (ADR-0067).
+- **`restore()` reads live rows only.** The store answers `WHERE status = 'live'`; terminal rows are not parsed at boot.
+- **A finished run's row lives 7 days, fixed.** Every non-live status — `done`, `failed`, `error`, `cancelled`, `lost`,
+  `drifted` — ages from its terminal write, and a sweep at boot and every hour deletes what is older. A `live` row is
+  never swept: a run parked on a Gate for a month stays and is restored. The value is fixed like the Sandbox idle
+  timeout, not config: jr2 is not the record of a run's results (a Machine sends its output where it belongs — a PR, a
+  Source), and `jr2 status` on an old run is a debugging act. A swept id frees its abbreviations (ADR-0009).
+- **The store is sqlite on the Instance's volume. No Postgres is owed.** The `SnapshotStore` interface stays as the
+  seam, with one implementation. `DATABASE_URL` is not a thing jr2 reads.
+
+## Considered options
+
+- **Keep `synchronous=FULL`, move the write to a worker thread.** Rejected: the thread hides 4.5 ms from the loop but
+  does not make the disk faster, so writes queue under load and the loop no longer knows how far behind the store is —
+  and it opens the ordering gap on every write, which NORMAL opens only on a power loss.
+- **Batch persists in a window.** Rejected: one persist per macrostep already coalesces a transition's events, and with
+  the fsync gone there is nothing left to amortize.
+- **A row per worker.** Rejected: it splits an atomic tree across rows (a worker's row saved, the Pool's not), makes
+  restore reassemble what xstate restores whole, and buys nothing measurable at 0.2 ms per blob.
+- **Configurable retention (`runs.retain`), or a count cap per workflow.** Rejected: a count cap does not bound a busy
+  workflow's disk and punishes a quiet one; a config key is owed only when a history surface (a list, a query) exists to
+  use it, and none does — `jr2 runs` and the Console list live runs.
+- **Build the Postgres store.** Rejected: the only thing it buys is an Orchestrator pod that reschedules without its
+  volume, which is a storage-class question (a network RWO volume follows the pod) and belongs to the Orchestrator pod's
+  own decision (R10), not to the store. Its cost is a dependency the owner operates, a second store in every test tier,
+  and the asynchronous write above.
+
+## Consequences
+
+- ADR-0008 and ADR-0009 lose "Postgres opt-in" and `DATABASE_URL`; ADR-0008's Orchestrator persists its snapshots "to
+  sqlite on its volume". ADR-0007 points here for the store's durability.
+- The Instance's volume gains `state.db-wal` and `state.db-shm` beside `state.db`.
+- A `drifted` run (ADR-0030) is swept a week after the deploy that changed its Machine, not a week after it ended; a
+  week is the time a human has to read it.