@jr2/orchestrator 0.4.1 → 0.6.0

package/docs/reference/decisions/0061-a-memory-kill-ends-the-agents-processes-not-the-harness.md

@@ -0,0 +1,83 @@
+# A memory kill ends the Agent's processes, not the Harness
+
+The Agent runs its programs (Playwright and its Chromium, `tsc`, `cargo`) through the `bash` Working tool, so they are
+children of the Harness, in the Harness container's one cgroup (ADR-0005, ADR-0037). On cgroup v2 the kubelet sets
+`memory.oom.group`, so when those programs pass the container's limit (ADR-0060) the kernel kills the whole container:
+the Harness dies with them, its conversation with it, and the Turn faults as "conversation lost" (lab 2026-09-28; chaos
+test T11). Worse, the bash tool that pi-agent-core 0.82.1 shipped reported a command killed by a signal as success
+(`exitCode: code ?? 0`), so when a program was killed the Agent saw its output stop partway and read it as a pass.
+
+**The Agent's processes** are every process in the Harness container except the Harness and its init: the tree of the
+running `bash` call, anything the Agent left running from earlier calls (a dev server, a watcher), and a human's
+`kubectl exec` session.
+
+## Decision
+
+Layers, each a backstop for the one before:
+
+1. **A process killed by a signal is an error.** jr2's bash tool reads the shell's exit: a signal death, of the command
+   or of the shell, reports the signal and never exit 0. A Harness conformance test holds it (ADR-0027).
+2. **`tini` is PID 1 in the Harness container**, shipped on `/opt/jr2`, so killed trees leave no zombies. Not
+   `shareProcessNamespace` — ADR-0005 keeps it off for good.
+3. **The `bash` Working tool starts each command with `oom_score_adj=1000`**, inherited by its children. It needs no
+   privilege, and on a node whose owner set the kubelet's `singleProcessOOMKill: true` the kernel then picks the Agent's
+   process, not the Harness.
+4. **The Harness guards its own cgroup.** Every 10–20 ms it reads anon + shmem from `memory.stat` against `memory.max`
+   (never `memory.current`, which counts reclaimable page cache). Near the limit it kills all of the Agent's processes,
+   checks usage again, clears `/dev/shm`, and the running `bash` call returns
+   `killed: memory limit (peak X of Y); use fewer workers or a larger Size`. The Agent hears it in the same Turn and
+   conversation, and can retry smaller. The guard polls from the Harness's start, not only during a call. A process that
+   an earlier Turn left running (a dev server, a watcher) can pass the limit while no Submission runs. For example, the
+   workflow waits at a human's Gate. Then no call returns the text. The Harness keeps every kill for the Orchestrator to
+   read, and a read removes nothing. The next Agent that works in that Workspace is told (ADR-0062).
+5. **The kernel's group kill is the last backstop.** The Orchestrator reads `lastState.terminated.reason: OOMKilled` and
+   the fault reason starts with the fixed prefix `memory limit`, not `Turn lost` (any other restart, ADR-0021) or
+   "conversation lost" (a 404). The fault reaches the Machine (ADR-0016); the next Agent in that Workspace is told
+   through the Briefing ([ADR-0062](0062-jr2-briefs-the-agent-on-its-seat-never-on-its-task.md)), never through the
+   Frame (ADR-0057). The kit's Machines pass the reason into their retry Turn as the example for authors. A kernel kill
+   while no Submission runs faults no Turn, because the restarted Harness rebuilds its conversations. The operator
+   publishes the container's last end on the Sandbox all the same, so before each admission into the Workspace the
+   Orchestrator reads it from the Sandbox watch, and a last end `OOMKilled` that no notice told yet tells the next Agent
+   (ADR-0062).
+
+The guard is best-effort and says so: a fast enough spike beats any poll. The lab won 35/35 on arm64 at 50 ms, 6/6 on
+amd64 at 10 ms (0.8% of a core), and only half on amd64 at 50 ms; it fired at 90–96% of the limit. Layer 5 covers what
+it misses.
+
+**The planned successor is a child cgroup for the Agent's processes** (`memory.max` below the container's, its own
+`oom.group`): exact, no race, the Harness always survives (lab: 3/3 on kind). It needs a writable cgroupfs, which today
+only a node owner grants (containerd ≥ 2.1 `cgroup_writable` behind a RuntimeClass, not offered on managed clusters).
+KEP-5474 adds `securityContext.cgroupOptions.mountMode: Writable` so the pod asks for it itself; it is alpha, targeted
+at v1.38. When it is enabled by default, jr2 sets the field on the Harness container and the child cgroup replaces layer
+4's polling.
+
+## Considered options
+
+- **Accept the group kill.** Rejected as the whole answer: every memory kill becomes a Turn fault, and the Agent never
+  learns why, so a retry repeats it.
+- **The Harness in its own container, with an exec server in the Sandbox Image's container running the Agent's
+  processes.** A real cgroup edge, but it reverses "Working tools execute in the Harness container" (ADR-0037) for a
+  problem layers 4 and the successor solve inside it.
+- **Adopt the child cgroup now, used when detected.** Rejected: a path almost no cluster runs is a path nobody tests,
+  and the guard must be solid on its own anyway.
+- **MemoryQoS `memory.high`.** Rejected: the lab saw a stall of over 9 minutes with no kill and `kubectl exec` hung.
+- **earlyoom, systemd-oomd, oomd.** Rejected: they read the host's `/proc/meminfo`, need systemd, or need PSI triggers,
+  which a read-only cgroupfs refuses. The guard borrows their idea.
+- **Kill only the running call's process group.** Rejected: `setsid` children escape it (lab), and a dev server left
+  from an earlier call may hold the memory.
+- **Raise Node's heap for the Agent's processes** (`NODE_OPTIONS`). Rejected in ADR-0060: it turns a readable V8 abort
+  into a kernel kill.
+
+## Consequences
+
+- The guard's threshold comes from the Harness container's limit, so it needs ADR-0060's limits first; layer 1 does not
+  and is fixed at once.
+- The kill set includes a human's exec session open at that moment; the group kill would end it too. On the Instance
+  Harness that session is the only process the guard can kill, because a Menu-only Agent has no Working tools. No notice
+  follows: that pod holds no Workspace (ADR-0031, ADR-0062).
+- A kernel kill (layer 5) while no Submission runs is told to the next Turn through the container's last end, which the
+  watch keeps until the container ends again. So two kernel kills before the next admission are told as one, and a
+  kernel kill that a later end of another kind replaces is not told. The guard fires at 90% of the limit, before the
+  kernel, so this window is the guard's misses alone.
+- A memory fault is named in `jr2 status` and on the feed, not `Turn lost` (a restart for any other reason) or
+  "conversation lost" (a 404).

package/docs/reference/decisions/0062-jr2-briefs-the-agent-on-its-seat-never-on-its-task.md

@@ -0,0 +1,239 @@
+# jr2 briefs the Agent on its seat, never on its task
+
+The model saw two sources of text: the Agent's `instructions` (identity, ADR-0018) and the Frame's prompt (the Machine
+state's, ADR-0057). jr2 itself spoke only after a failure — the no-signal nudge — so an Agent had no way to learn facts
+only the mechanism knows. ADR-0057 opens with that failure: a weak model never learned its working directory, wrote
+outside the worktree and still called `finish`. ADR-0060 adds more such facts: `os.cpus()` reports the node, not the
+Sandbox's CPUs. ADR-0061 adds events: a kernel memory kill ends a conversation, and the next Agent inherits a Workspace
+whose processes are gone, with no way to know why.
+
+## Decision
+
+- **jr2 has its own text to the model, the Briefing.** It says a fact only if a correct Agent would act differently
+  knowing it **and** cannot find it out for itself: how it acts on the workflow, the Sandbox's CPUs and Size, the
+  working directory, that a Menu-only Agent has no Working tools, and what happened to the last Turn. Nothing about the
+  task — that is the Frame's — and nothing about style or care — that is identity's. Stand-ins are left out: the only
+  change that knowing could cause is an attempt to go around the Custodian.
+- **Two parts, placed by how often they change.** The standing part is the same on every Turn of a seat; it follows
+  `instructions` in the system prompt, in its own delimited section, and is byte-stable (no timestamps, no counters, no
+  Menu names). The Turn part follows the Frame's prompt in a delimited block: the working directory, the notices, a
+  **pick's briefing** for each Allowed pick that has one (jr2's text on the pick's parameter, which the Orchestrator's
+  surface marks on the pick: the Chart briefing of [ADR-0070](0070-a-machine-lends-its-parts-to-a-chart.md), said only
+  on a Turn that may make that pick), and the Turn's Allowed picks
+  ([ADR-0029](0029-a-menu-is-fixed-for-a-conversation-and-a-turn-is-told-its-allowed-picks.md)). So the Briefing never
+  breaks the provider's prompt cache — the Turn part sits in the newest message, which is uncached anyway, and stays
+  unchanged in the history after.
+- **How the Agent acts on the workflow is one standing statement**, true whether a pick ends the Turn or keeps it open
+  ([ADR-0024](0024-an-agents-turn-ends-with-the-state-that-asked-for-it.md)): "You act on the workflow by picking from
+  your Menu: a call to one of the workflow's tools that the newest "Allowed now" lists. Each pick's answer says if your
+  Turn is over. If it is over, stop and call no other tool. If it is not, continue. If nothing is allowed, stop: the
+  workflow prompts you again when something is." The last sentence is safe because the wake asks a model that stopped
+  with nothing allowed when a pick opens ([ADR-0016](0016-agent-turn-mechanics-are-internal.md)). A line that says a
+  Turn ends with exactly one pick would be false for a pick that keeps the Turn open.
+- **The Turn part is written once per Submission, and restated when its Allowed picks change.** When something other
+  than the Agent's own pick changes them inside the Turn, the Harness appends the Allowed picks again at the next step
+  boundary, in a `<jr2-turn>` block of their own: "The workflow's state changed since you last read the Allowed picks.
+  Allowed now: …" ([ADR-0029](0029-a-menu-is-fixed-for-a-conversation-and-a-turn-is-told-its-allowed-picks.md)). It is
+  appended, never a rewrite of the first, so the cache argument above holds for it too. It carries a pick's briefing
+  when the restatement opens that pick. The same read at a step boundary returns the conversation's pending notices, and
+  the Harness appends them in the same way (below).
+- **The Turn part comes last, because a model obeys what it read last.** Measured on the home-lab vLLM model: with a
+  prompt that argued for a pick the Turn did not allow, the Allowed picks placed ahead of the prompt were obeyed 0/10
+  (thinking off) and 5/10 (thinking on); placed after it, 10/10 both. Wording made no difference. The author's task
+  still reads first, as it would with no jr2.
+- **Always on.** No author can turn it off: every fact in it passes the rule above, so removing one can only make the
+  Agent act worse. An `instructions` that repeats a fact is harmless.
+- **A notice is heard once by each member of its audience.** A member takes it at its next admission, or at a step
+  boundary of an open Turn: the surface read of the open Turn returns the conversation's pending notices, and the
+  Harness appends them, as it appends a changed Allowed-picks list
+  ([ADR-0029](0029-a-menu-is-fixed-for-a-conversation-and-a-turn-is-told-its-allowed-picks.md), ADR-0002). For a guard
+  kill that the Agent of an open Turn was not told, a member also takes it at that Turn's nudge or wake (below). A
+  conversation is an Instance ID ([ADR-0057](0057-a-turn-is-its-frame-its-dials-and-whether-it-continues.md)): a fresh
+  Turn, a runaway reroll and a burned conversation's successor each start a new one.
+  - **Conversation scope** ("this conversation is new; earlier context is gone") goes to that Agent's next Turn. So does
+    a pick's answer that never reached its call
+    ([ADR-0024](0024-an-agents-turn-ends-with-the-state-that-asked-for-it.md)): the route left the Agent's state, and
+    the next Turn of the conversation the picking Turn continued hears "the workflow left the state of your last turn
+    while your pick was outstanding", with the answer — or, when something other than the route left the state while the
+    pick's work ran, "your pick did not finish: the workflow left the state that asked for your last turn while it ran,
+    and that stopped it" (`stopped`), with an earlier link's reply when one came. No pick-answer notice says the work
+    ran to its end. A fresh picking Turn has no such conversation, and its answer is no notice.
+  - **Workspace scope** ("the Sandbox's processes were killed at its memory limit during `coder`'s Turn; `/dev/shm` was
+    cleared. Anything that was running in the Sandbox at that time is gone.") goes to an **audience** that the ledger
+    fixes when it raises the notice. The rule is: an Agent hears a kill if it was there for it.
+    1. Each conversation admitted in that Workspace before the kill that can still take a Turn: the conversation of an
+       open Turn, which hears it at its next step boundary, or a continued conversation, which hears it at its next
+       Turn. A fresh Turn that already ended is not in the audience.
+    2. When the kill cut a specific Turn — a kill read on that Turn's stream names it — that Agent's next Turn, fresh or
+       continued: the retry of the broken work.
+    3. Minus each conversation whose own `bash` answer returned the kill's verdict (`told`, below). That answer told it
+       already.
+
+    Each member hears the notice once. The notice retires when every member has heard it. A conversation that starts
+    after the kill does not hear it, except the retry of (2). So a Pool of coders in one Workspace
+    ([ADR-0075](0075-a-machine-cuts-a-worktree-and-the-workspace-only-clones.md)) all hear a kill that cut their work.
+    Only a kill read on a stream while its wait is live names the Turn whose Submission ran. Both forms end with the
+    whole Sandbox, because every process in it died, a shared Workspace's too, and in the past tense, because the Agent
+    may hear it after processes started again. A kill read from the pod's list names no Turn and says nothing of when or
+    whose: "the Sandbox's processes were killed at its memory limit (peak X of Y); /dev/shm was cleared. Anything that
+    was running in the Sandbox at that time is gone." The list cannot say whose Turn a kill ended: the kill may land
+    after a state exits, or end an attach, a sync or a command. A kernel kill read from the Sandbox's last end (below)
+    says the same, with the limit and no peak: "the Sandbox's processes were killed at its memory limit of Y; …". A
+    Menu-only Agent outside a Workspace receives conversation notices only.
+
+  - A notice never replaces a fault. The Machine still receives `agent.fault` and decides (ADR-0016); the notice makes
+    sure the next Agent is not working blind.
+
+- **The Orchestrator is the only keeper of notices**: pending notices are plain data in the run's persisted state
+  (ADR-0007). It sees kernel kills and fresh conversations itself. A kernel kill (ADR-0061, layer 5) has two paths,
+  under one source, the Workspace and the time the Harness container ended. A Turn that the kill ended faults on it, and
+  the fault raises the notice with the Turn's Agent. A kill while no Turn runs faults nothing, so before each admission
+  into a Workspace the Agent actor also reads the container's last end from the Sandbox watch, which the operator
+  publishes: a last end `OOMKilled` raises the notice, which names no Turn. The ledger's `seen` makes the two paths one
+  notice. An actor whose own watch saw the restart reads the last end only when it re-admits the Turn after the cut.
+  When the Turn faults, the fault's notice names the Turn's Agent. A watch that has not listed yet says nothing, and the
+  next admission reads again: the last end stays until the container ends again. A nudge and a wake take no kernel kill;
+  it stays for the next Turn. The re-admission after a cut takes it, as it takes every notice in scope that is not a
+  guard kill. The Harness reports a guard kill (ADR-0061) on two paths, under one id that the Harness gives the kill:
+  - **On a stream.** While a Submission runs, the kill is an event on that conversation's updates stream. The Agent
+    actor reads a stream while it waits on the Submission. It stops when the Agent's state exits, before its abort
+    reaches the Harness. So a stream does not always reach a reader: a kill in that window is on a stream that no one
+    reads. A stopped actor raises nothing from its stream.
+  - **In the pod's list.** The Harness keeps every kill, with `told`: the conversations whose `bash` call returned the
+    kill's verdict to the model. A conversation goes into it when the answer returns, not when the guard fires: a call
+    cut short (the container's end, a restart) told no one. A model that writes text, or waits, is not told. The Harness
+    keeps the list in a file beside its conversations and answers it on `GET /memory-kills`. A kill while no Submission
+    runs is on this path alone: for example, a dev server killed while the workflow waits at a Gate. Before each
+    admission into a Workspace, the Agent actor reads the list. The actor sets the read's limit of 5 seconds, and it is
+    the only limit. A read that never left the Orchestrator, or a draining Harness's 503, is sent again in that time. A
+    read that fails or times out is none, and the next admission reads again: the list keeps the kill. The Orchestrator
+    logs one line when the reads from a Harness start to fail. It logs no line when a read works again, and it logs the
+    failure again only when a read fails after a read that worked. The file is beside the Harness's conversations, which
+    no confined child can reach
+    ([ADR-0076](0076-an-agents-control-credential-lives-only-where-no-workspace-code-runs.md)). The bounds stay as a
+    second layer: a `peak` or a `limit` that is not a size in the guard's form, or is long, is left out and logged, on
+    the Harness's read of the file and on the Orchestrator's read of the list and of a stream. The kill is kept, so a
+    real kill is never lost, and its notice says no sizes: "the Sandbox's processes were killed at its memory limit;
+    /dev/shm was cleared. …". Only a kill with no id or no time, or one outside the bounds below, is dropped. A stream
+    event from an older Harness has no id; its source is then its place on the stream, exactly as the Orchestrator
+    raised it before.
+
+  Both paths raise the notice with the same source, the Workspace and the kill's id. The ledger keeps every source it
+  raised (`seen`) in the run's persisted state. So a kill read on both paths, or on two streams, is one notice. Each
+  read of the list raises one notice at most: the newest kill's, unless it is raised already. The read marks the older
+  kills in it seen, and they give no notice, so a burst of kills in one read tells the Agent one thing. Kills read on a
+  stream are not folded: each one with different sizes gives its own notice. Two notices with the same words keep two
+  entries in the ledger, and an admission that takes both says them once. The run is saved when a raise changes the
+  ledger, and only then. A read removes nothing, so a crash between the read and the save that keeps it loses no kill. A
+  restart of the Orchestrator raises no kill twice. A restart of the Harness container keeps the file, which is on the
+  pod's emptyDir with its conversations (ADR-0031). A lost pod takes the file with it, but then the Workspace is lost
+  (ADR-0021). The list keeps the newest 16 kills, each with at most 32 conversations in `told`, and the Orchestrator
+  reads at most 256 KiB of it. The two sides hold the same bounds: an `id` and an `at` of at most 64, an `agent` and an
+  `instanceId` of at most 128, and an `at` that parses as a time. A kill outside them is dropped, and a `told` entry
+  outside them is left out. The Harness counts the bytes of the JSON text, and the Orchestrator counts characters, which
+  are never more, so the Harness keeps no kill that the Orchestrator drops. The Harness reads at most the last 256 KiB
+  of the file. With these bounds the largest list is 149,259 bytes. When more come between two admissions, the oldest is
+  lost, and a newer kill tells the same thing. When more than 16 come in one Turn, the Turn's mark leaves the list too,
+  and every kill in the list then counts as after the mark; a nudge or a wake of that Turn may then carry a kill from
+  before it. The Instance Harness keeps no kill, because it holds no Workspace.
+
+  A notice counts as delivered when its admission is ledgered, so a restart before that point delivers it again and a
+  restart after it does not. A notice at a step boundary has no admission. The surface read of the open Turn reserves it
+  for that member, and it counts as delivered, and heard, when the Harness acknowledges the append. When the Turn ends
+  with no acknowledgment, the reservation is released, and the notice is pending again for that member. A reservation
+  lives in the Orchestrator's memory only, so a restart forgets it: the notice is pending again, and the next read or
+  admission carries it. So a conversation can hear a notice twice after a crash, as for an admission. Delivered is not
+  yet heard: if the Orchestrator ends the Turn before the Harness reads its surface, the Harness never prompts it
+  (ADR-0026), so the Orchestrator raises that admission's notices again. This holds for a nudge and a wake too: each one
+  must have its surface read again before its notices count as heard. An admission that fails leaves the earlier
+  admission's notices heard. A nudge and a wake re-prompt the Turn within itself. Of the notices, they take only the
+  guard kills of the list that come after the newest kill whose `told` names their conversation. That `bash` answer told
+  the Agent that all its processes died, so it covers every kill up to it. They also take no kill from before the Turn
+  started. Such a kill would tell the Agent that processes it started in this Turn died. The Turn's first read of the
+  list sets its mark: the id of the newest kill it saw, or "none". The mark is ledgered with every admission of the
+  Turn, and a restore reads it back, so a nudge or a wake takes only kills after it. Only when the first read failed is
+  the Turn's start the mark, ledgered the same way, with one second allowed for a pod clock that runs behind. A read
+  that failed is "could not tell", never an empty list, so it sets no mark. The kills they do not take stay pending for
+  the next Turn. The re-admission after a Harness restart cut the Turn continues the same Turn, so it takes every notice
+  in scope except the guard kills that this rule leaves out. When its read could not tell, it takes no guard kill at
+  all, as a nudge then takes none: it cannot know which kills its own `bash` reported. The notices that the Turn's
+  admissions carried since the Harness last read the Turn's surface are pending again first, each with the source it was
+  raised with, and this re-admission carries them whatever else its scope leaves out. A notice with no source merges
+  only into a pending one with no source, so an equal notice of another kill never takes its place. When such an unread
+  admission carried the Turn's task, the task follows the re-admit text. When the rebuilt Harness lost the conversation,
+  nothing of the Turn is left: every notice that the Turn's admissions carried is pending again whatever the surface
+  reads said, the re-admission takes every notice in scope, as a Turn's admission does, and the Turn's task follows the
+  re-admit text. It says once that the conversation is new. A restore drops the prompt from a Turn's input, so the task
+  rides the ledgered admission too; a lost conversation with no task to send faults the Turn instead. Whether the
+  Harness still holds the conversation is asked again for up to 30 seconds while the new container does not listen yet
+  or drains, and only then is the answer taken as "held". The continue path asks the same way, so a continued Turn after
+  a restart can wait up to 30 seconds before its admission. A Harness in CrashLoopBackOff can take longer than that: the
+  answer is then "held", and it can be wrong. So every admission to a conversation that the Harness was taken to keep
+  says so with `kept: true`: the re-admission after a cut, a continued Turn, and a nudge or a wake. A Harness that keeps
+  no such conversation refuses it with 404 and `kept: false` in the body, and seats none. The Orchestrator takes only
+  that 404 as the refusal; another 404 is a failed admission. Without the refusal the Harness would seat a new
+  conversation, and a model on an empty context would read "continue your task" with no task. On the refusal the Turn
+  takes the path of a lost conversation. The re-admission is admitted as a Turn, with the task after the re-admit text
+  and the notice that the conversation is new, or faults when it has no task to send. A continued Turn is told that its
+  conversation is new, and is admitted again. A nudge or a wake is admitted again as a Turn: its prompt is the task
+  alone, because nothing was cut, with the notice that the conversation is new and every notice in scope. Every notice
+  that the Turn's admissions carried is pending again first, as for a lost conversation after a cut. With no task to
+  send, the Turn faults. The Instance Harness's restart is on no watch, so this refusal is the only way a nudge or a
+  wake learns of it. What the Turn's admissions carried, and whether the Harness read their surface, is kept in the
+  Orchestrator's memory alone: after a restore of the Orchestrator before a surface read, nothing is raised again. A
+  fresh Turn's "this conversation is new", which rides an admission alone, is carried again by the next re-admission
+  when the Harness has not read the one before.
+
+  A conversation whose own `bash` call answered a kill is not in that kill's audience, so it does not hear the kill
+  again as a notice, at its next Turn or at a step boundary.
+
+- **The Harness writes every word.** The admit body carries typed notices
+  (`notices: [{ kind: "memory-limit", scope, agent?, limit?, peak? } | { kind: "conversation-new", reason } | { kind: "pick-answer", pick, cut, answer?, stopped? } …]`,
+  where `peak` comes only with `limit`), and the Harness renders both parts. An outstanding pick's answer is data too —
+  the last reply since the pick, a party's event with its payload or a child's output (ADR-0024) — and its words ("Its
+  answer: `approve` from Gate `review`, with {…}", "No reply came back for it.") are the Harness's, in the receipt and
+  in the notice alike; "it moved the workflow, and nothing ran" is a guard straight back's receipt alone. The standing
+  part needs no wire: the Harness already holds its limits (cgroup and Downward API env), the definition's `workspace`
+  access, and the Menu. The wording lives in one place, and Harness conformance checks it (ADR-0027).
+
+## Considered options
+
+- **Put the facts in the Frame.** Rejected: the Frame is the Machine state's (ADR-0057), and a kit that writes into it
+  makes the author's prompt say things the author did not write.
+- **Leave it to authors' `instructions`.** Rejected: a definition is written before any run, so it cannot know a Size a
+  composer retuned, a working directory, or what happened to the last Turn — and packaged Agents would each word it
+  differently or not at all.
+- **An author switch** (`agent({ briefing: false })`, or notices only). Rejected: under the rule, each fact is one a
+  correct Agent needs; the switch only lets the ADR-0057 failure back in.
+- **Everything in the system prompt.** Rejected: per-Turn content there breaks the prompt cache on every Turn.
+- **Finished text from the Orchestrator.** Rejected: wording split across two processes, and untestable where the turn
+  loop is tested.
+- **Report a guard kill on the stream alone.** Rejected: a kill while no Submission runs, or after the wait stopped,
+  reaches no reader. A dev server killed while the workflow waits at a Gate is then never told to the next Turn, which
+  works blind.
+- **Put that kill on every conversation's stream.** Rejected: the next Admission reads its conversation's stream from
+  its own offset, which is after the event.
+- **Keep in the list only the kills that no stream carried.** Rejected: a kill on a stream whose wait stopped is then in
+  no place that is read. And two streams that carry one kill would give two notices.
+- **The Harness pushes the kill to the Orchestrator.** Rejected: the Harness container is the Agent's, and ADR-0013
+  closes a route from it to the Orchestrator. A push also needs a retry and an acknowledgement, and an idempotent read
+  does not.
+- **Read the kill on the Sandbox watch.** Rejected: the operator publishes what the kubelet says about the container,
+  and a guard kill does not end the container. The Harness has no Kubernetes credentials to publish it.
+- **The Harness words the kill into the next Turn itself.** Rejected: a second keeper of notices, outside the run's
+  persisted state, which cannot know when an admission is ledgered or heard.
+
+## Consequences
+
+- ADR-0060's CPU fact and ADR-0061's memory-limit notice are the Briefing's first users.
+- Each admission into a Workspace costs one more request to its Harness, `GET /memory-kills`. The first read from a
+  Harness that fails is said in one line of the Orchestrator's log.
+- The no-signal nudge stays a re-prompt within a Turn (ADR-0016); it is not a notice.
+- The Briefing is how a Turn learns its Allowed picks: the tools block is the whole Menu, fixed for a conversation so it
+  does not break the cache
+  ([ADR-0029](0029-a-menu-is-fixed-for-a-conversation-and-a-turn-is-told-its-allowed-picks.md)).
+- A Workspace notice reaches each conversation that was there for the kill, and the retry of the Turn that it cut; an
+  open Turn hears it at its next step boundary, not only at its next admission (ADR-0002,
+  [ADR-0029](0029-a-menu-is-fixed-for-a-conversation-and-a-turn-is-told-its-allowed-picks.md)). So a memory kill reaches
+  every coder of a Pool that shares one Workspace (ADR-0075).
+- GLOSSARY.md gains **Briefing**.

package/docs/reference/decisions/0063-the-orchestrator-watches-the-cluster.md

@@ -0,0 +1,82 @@
+# The Orchestrator watches the cluster
+
+The Orchestrator talked to Kubernetes by starting a `kubectl` process for every call (`sandbox-kubectl.ts`). A short
+process can only ask "what is the state now?", so waiting became polling: `get sandbox` every second while a Workspace
+provisioned (and `get pod` every fifth time), `get sandbox` every second while a fetch ask waited (ADR-0053), and
+detection of a lost Workspace only on the Lease's 5-minute renewal (ADR-0021 chose that and named the watch as the thing
+to revisit "if something needs sub-minute loss detection"). The attach was a `kubectl exec` stream per Workspace through
+the API server. The scaling review (2026-09-27) measured 36 concurrent `kubectl` processes and about one core of
+Orchestrator CPU at 40 Workspaces starting, and `workspace.lost` arriving 4.6–4.7 minutes after an eviction or drain.
+Decisions since then need pod facts at once: the scheduler's reason for a Pending pod, a lost pod's reason, and
+`OOMKilled` to name a memory fault (ADR-0060, ADR-0061).
+
+## Decision
+
+- **One in-process client on Node's built-in `fetch`, with no new dependency.** The Orchestrator is always in-cluster
+  (ADR-0019). The Deployment sets `NODE_EXTRA_CA_CERTS` to the ServiceAccount's `ca.crt` (concatenated with the ADR-0020
+  bundle if the Orchestrator ever needs both). The client does plain REST for jr2's own kinds — Sandbox, Repo, Secret —
+  including server-side apply, and one watch loop. Its rules, each observed on a real cluster (2026-09-29):
+  - The token is read from its file on every request, never cached: the kubelet replaced a 10-minute token only about 50
+    seconds before it expired. A `401` re-reads the file and retries once.
+  - A watch asks for bookmarks and `timeoutSeconds=240`. Built-in `fetch` aborts a response body that sends nothing for
+    300 seconds (`UND_ERR_BODY_TIMEOUT`, at 300.8s), and that timeout cannot be changed without undici's dispatcher; the
+    server closing first, and bookmarks about every 60 seconds, keep a healthy watch clear of it. The abort itself is a
+    resume, not a failure.
+  - Every bookmark's `resourceVersion` is kept, so a resume after a quiet period starts from a recent point.
+  - A resume that is too old arrives as an `ERROR` event with code 410 **inside an HTTP 200**. Any `ERROR` event
+    re-lists.
+  - Concurrent writes are capped (8–16). 200 parallel writes each opened a TLS connection and cost about 130% CPU.
+- **The Orchestrator watches Sandboxes only**, one watch on the Instance's namespace — the namespace is the Instance's
+  identity (ADR-0019), and `jr2 up` refuses one another Instance owns — selected by the `jr2.dev/run` label every
+  Sandbox it provisions carries. Observed: event latency p50 11ms, p95 92ms, max 186ms; 0 events lost across a
+  200-object burst and a CRD run; about 75 MiB and 3% CPU for the watcher. **It never reads a Pod.** The operator owns
+  the pod (ADR-0001) and already reads it on every reconcile, so it publishes what the Orchestrator needs onto the
+  Sandbox's status: a scheduling condition with the scheduler's message — or a quota's refusal of the pod create
+  ([ADR-0064](0064-a-workspace-waits-for-capacity.md)) — the Harness container's restarts and last terminated reason,
+  and `podUID`.
+- **The watch drives everything that polled.**
+  - Placing and provisioning wait on watch events: an Unschedulable pod or a quota refusal is a wait with its reason
+    (ADR-0064), a crash-looping Harness is seen as it happens — and named: the Harness container's termination message
+    falls back to its last log lines (`terminationMessagePolicy: FallbackToLogsOnError`), so the fault that ends the
+    provision carries the Harness's own last words, and the Harness says them plainly at startup (a port another
+    container in the pod took: `EADDRINUSE` names the port and the cause). No reason is invented for what a log line
+    already says. And readiness proves the Harness, not the port: the probe is a GET of the Harness's own `/healthz`,
+    because a pod has one network namespace and a socket probe passes for whatever listens there — the nginx that took
+    `:8080` made the pod Ready and the attach met its 404 page (scaling review R16, 2026-09-30). A pod with a Sandbox
+    Effector is Ready only when that Effector serves too: its container is probed on its own `/healthz` (ADR-0068).
+  - Continuity: a gone or `Lost` Sandbox is `workspace.lost` within seconds, and a Harness restart ends the Turns
+    waiting on it (ADR-0021). A dropped watch is unknown, never loss; the loop re-lists and reconciles.
+  - A fetch ask (ADR-0053) is answered by the watch event that carries its landing.
+  - Restore (ADR-0012): the first list after start is the reconcile.
+- **The Lease stays a write.** It is still the assertion that lets the operator reap an orphaned Sandbox (ADR-0001): one
+  merge patch per Workspace every 5 minutes, ±20% jitter so Leases never renew in lockstep after a restart. It no longer
+  carries detection.
+- **The attach is a Harness call, not an exec.** The Harness runs the attach — ACL stamp, `git clone --shared` from the
+  node cache, remotes — on `POST /attach { slots }`, authenticated like every Harness call (ADR-0058). No stream per
+  Workspace crosses the API server, and **`pods/exec` leaves the Orchestrator's RBAC**. It also binds work late: a pod
+  already running receives its Repos, the shape standby Sandboxes and Agent Substrate need.
+
+## Considered options
+
+- **`@kubernetes/client-node`.** Informers built in, but a large dependency tree and generated models for every kind jr2
+  never touches, around kubeconfig handling in-cluster code does not need. A contained swap if jr2 ever needs exec or
+  many more kinds: every call stays behind the Sandbox and Repo ports.
+- **`node:https` with its own agent.** Trust scoped to this client alone, but transport code built-in `fetch` makes
+  unnecessary; the Orchestrator sets `NODE_EXTRA_CA_CERTS` for nothing else.
+- **Keep `kubectl`, add `kubectl get -w`.** Fixes detection, not the process per write, and adds a long-lived child per
+  watch.
+- **Watch Pods as well.** Two watches, and the Orchestrator interpreting pod internals the operator already reads.
+- **A jr2 WebSocket API.** Nothing to add: a Kubernetes watch is already a pushed stream of events.
+- **Keep the attach as exec through the new client.** The exec protocol is the hardest part of any Kubernetes client, it
+  keeps one stream per Workspace through the API server, and it keeps `pods/exec`.
+- **Attach from an init container.** Declarative, but it fixes the Repos at pod creation, which rules out late binding,
+  and it must wait on a cache that exists only after scheduling.
+
+## Consequences
+
+- ADR-0021 is rewritten in place: the watch answers, the Lease only asserts. ADR-0012, ADR-0004 and ADR-0053 now read
+  the watch and the Harness attach. ADR-0001: the operator publishes pod facts on the Sandbox's status, and writes
+  status only on change — every write is now a watch event at the Orchestrator.
+- The Harness gains a route that is not about a conversation. It is jr2's own work in the container ADR-0005 gives it.
+- The CLI keeps `kubectl` on the user's machine (`jr2 up`, `jr2 ssh`, ADR-0009); only the Orchestrator stops using it.
+- Tests fake the client with a small HTTP server instead of a `kubectl` stub.

package/docs/reference/decisions/0064-a-workspace-waits-for-capacity.md

@@ -0,0 +1,79 @@
+# A Workspace waits for capacity
+
+Since ADR-0060 a Sandbox reserves its whole Size, so a full cluster shows as a Pending pod, not as eviction. But the
+provision's 120-second budget started at the CR's write, so time spent Unschedulable used it up: the run errored, and
+the CR, pod and Secret stayed until the operator's idle GC. A burst of 10 Workspaces at 500m each on the home cluster
+(2026-09-29) placed 2; the other 8 errored at 120s and left Pending pods behind, one of which could take a node later
+for nobody. Worse, the Lease was renewed only in `running`, so the operator counted creation as the Lease and would reap
+a Sandbox that waited past its 30-minute idle timeout. The scaling review asked two more questions: should jr2 cap live
+Workspaces for the Instance (R4), and concurrent Turns per model provider (R5)?
+
+## Decision
+
+- **A Sandbox with no node waits, and jr2 sets no deadline on the wait.** jr2 cannot tell "full now" from "never fits":
+  the cluster autoscaler, a finished neighbour or a drain can each make room, and the node set moves (ADR-0052).
+  `jr2 up` already warns when no node could hold a Size (ADR-0060). A Machine that wants a bound puts `after` on the
+  state that holds the Workspace; stopping it is the body's policy, as for `workspace.lost` (ADR-0021).
+- **A quota refusal is the same wait.** A ResourceQuota (or Kueue) is the cluster owner's tool; jr2 neither writes one
+  nor replaces it. The operator publishes the refusal on the Sandbox's status and keeps retrying the pod create.
+- **The Workspace Machine has a state for the wait: `placing`**, before `provisioning`. `placing` writes the CR and
+  waits for the pod to be scheduled, with no deadline. `provisioning` waits for `Ready` under the 120-second budget,
+  which now measures only what jr2 controls: the image pull, the Harness start and the Repos. A restart re-enters
+  `placing`: the apply is idempotent and the watch's first list answers at once (ADR-0063).
+- **The wait is visible; its reason is behind the token.**
+  - The state is structure, so the Console lights `placing` and `observe()` carries it with no new field.
+  - The authenticated `RunStatus` gains `waiting: [{ child, on: "node" | "quota", message, since }]`. `message` is the
+    scheduler's or the quota's own words, which can name nodes and taints. `jr2 status <run>` prints it.
+  - The feed has two lines per wait: when it starts, with the reason, and when it ends ("placed after 3m12s"). The
+    duration of a wait this Orchestrator process opened is read on the monotonic clock, so it is never negative. A wait
+    opened before an Orchestrator restart has only its stored wall-clock start, so its duration is two wall-clock reads,
+    clamped at 0. A changed reason updates `waiting`; it adds no feed line.
+  - `jr2 runs` marks a run that has a Workspace in `placing`.
+  - The Machine receives no event: scheduling detail is not the body's concern, and `after` covers a bound.
+- **The Lease covers the Sandbox from its write to its teardown**, in every state, so the idle GC reaps only what the
+  Orchestrator has abandoned (ADR-0001).
+- **Delete what never ran; keep what ran.** A Sandbox whose wait ends without a node — a parent `after`, a stopped run,
+  a faulted run — is deleted at once: nothing on it can be inspected, and left alone it could take a node later. The
+  delete does not touch a name that a live Workspace of the run holds then: a parent that re-enters the state, or
+  invokes the same Machine again, gives its new Workspace the same invoke id and so the same name, and that Workspace
+  has placed it again ([ADR-0073](0073-a-machine-fails-a-run-with-fail-and-an-uncaught-failure-ends-it-failed.md)). A
+  Sandbox that was placed and then faulted keeps ADR-0021's rule: it stays inspectable (logs, `jr2 ssh`) and ages out at
+  the idle timeout.
+- **jr2 caps nothing of its own.** Not live Workspaces (R4), not concurrent Turns per provider (R5). See the options.
+- **A provider's refusal is named.** A Turn that ends because the Harness's provider retries ran out has a fault reason
+  with the fixed prefix `provider limit`, as a memory kill has `memory limit` (ADR-0061), so `jr2 status` and the
+  Machine can tell provider pressure from a bug.
+
+## Considered options
+
+- **Keep a deadline on the wait, or make it a config value.** Rejected: any number either faults a Workspace the
+  autoscaler was about to place or waits for one that never fits, and the Machine's own `after` already states the
+  body's bound where the body can act on it.
+- **An Instance-wide cap on live Workspaces (`sandbox.max`)**, first come first served, persisted: Agent Substrate's
+  request parking in jr2's terms. Rejected: a count of Workspaces of mixed Sizes is not a measure of resources. A spend
+  ceiling is a ResourceQuota on `requests.cpu` and `requests.memory`, exact in resource units and refused before any pod
+  exists, so the autoscaler never scales for it. Placement order among Pending pods of one priority is not promised, and
+  nothing in jr2 needs strict order. Substrate's parking is driven by capacity, not a count.
+- **An Instance-wide cap on concurrent Turns per provider (R5).** Rejected:
+  - A Turn is the wrong unit: it is many model calls with tool runs between them, and a slot held while `tsc` runs
+    blocks the provider while it has nothing to do.
+  - A fixed number is the wrong measure: hosted providers limit tokens and requests per minute, vLLM by batch and KV
+    cache.
+  - The wait already happens in the right place: vLLM queues in arrival order, and the Harness retries twice, through
+    pi-ai, with backoff, and honors `retry-after` up to 60 s (pi-ai 1.4.0). pi-ai retries what the SDKs retry: a 408, a
+    409, a 429, each 5xx, and a connection error (an error with no status). So a quota 429 is asked again too; its
+    settlement keeps the quota's own words, never `provider limit`. The scaling review measured slow Turns at 40
+    concurrent (about 90s against 14s alone), which is throughput a cap cannot add, and no failed ones.
+  - A slow provider cannot make a Runaway: ADR-0035 counts steps and repeats, not time.
+  - If a hosted provider's 429 storm is ever measured, the answer is shared backoff at the Custodian (Envoy's global
+    rate limit), not a Turn count.
+- **Watch for a Pending pod from the Orchestrator.** Rejected by ADR-0063: the operator reads the pod and publishes the
+  scheduling condition.
+
+## Consequences
+
+- ADR-0012's lifecycle gains `placing`. ADR-0021: the Lease is no longer only in `running`, so "attach must stay well
+  inside the idle timeout" goes away. ADR-0001: the operator publishes a quota refusal like an Unschedulable pod.
+- A Workspace parked in `placing` holds no node, only a CR and a Secret. A Workspace that waits a long time still holds
+  its run's place in any `pool()`: the Pool's cap is the Machine's own concurrency policy.
+- GLOSSARY.md gains **Placing**, and **Lease** covers the Sandbox's whole life.

package/docs/reference/decisions/0065-a-persist-survives-a-crash-and-history-is-a-week.md

@@ -0,0 +1,65 @@
+# A persist survives a crash, and history is a week
+
+Every transition of a run persists its whole snapshot (ADR-0007). The scaling review (R6) read the store and listed its
+costs: `JSON.stringify` of the whole run, a synchronous upsert with an fsync on the event loop, a Pool's run as one blob
+holding every worker, terminal rows never deleted, and `restore()` parsing every row before the server listens. It also
+found the words "Postgres opt-in" in GLOSSARY.md, ADR-0008 and ADR-0009, with no such store in the code. Measured on the
+home cluster's chaos Instance (2026-09-30, `local-path` volume): a persist costs **4.5 ms, whatever the blob's size** (1
+KB or 100 KB), because the cost is the fsync, not the JSON; with the write-ahead log and `synchronous=NORMAL` the same
+upserts cost 0.04 ms (1 KB) and 0.18 ms (100 KB). A run's blob is about 1 KB; a Pool of 40 workers is about 100 KB.
+
+## Decision
+
+- **A persist survives a process crash, not a power loss.** The store runs sqlite in WAL mode with `synchronous=NORMAL`:
+  a write is durable once the OS holds it, and sqlite fsyncs only at a checkpoint. A node's power loss can lose the
+  transitions since the last checkpoint; the file stays consistent, and `restore()` treats the older snapshot as it
+  treats any restart — reconcile against the live world, re-attach, or `lost`. The window in which a crash re-POSTs a
+  prompt (ADR-0007's at-most-once sliver) widens from one microtask to one checkpoint, and only on a power loss, which
+  also ends every Sandbox on that node.
+- **The write stays synchronous and on the event loop.** At 0.2 ms for a Pool's blob there is nothing to move off the
+  loop, and the ordering matters: the row is on disk before the Machine's next side effect runs, which is what lets an
+  epoch bump (ADR-0057), an admission (ADR-0016) or an Effect's intent row (ADR-0067) be saved "at the write" — the
+  intent row before its call leaves. A worker thread, an asynchronous driver or a network store would land the row after
+  the next `invoke` fires, on every persist.
+- **One blob per run stays.** The snapshot is a tree that xstate restores from its root, and one row keeps it atomic —
+  with the ledgers beside it (admissions, epochs, notices, timers, outstanding picks, Effects); a Pool's cost at 40
+  workers is 0.2 ms, so its workers do not become rows of their own. The Effect ledger keeps a row for every Effect the
+  run started (ADR-0067), so a Machine that loops through an Effect grows its blob by one row per pass: id, kind and
+  outcome. The output's tail is dropped once the save that carries the Effect's transition is on disk; a push row keeps
+  its published commit, its source, its `squashed` and its sync's answer whole — every later push of its Repo Slot sends
+  them as one of its pairs — and its bundle only while it is its Repo Slot's newest push and its sync did not finish —
+  one bundle per Repo Slot at most. An outstanding pick's hold is the one place tails accumulate: its `chain` keeps
+  every reply since the pick whole until the pick is answered, then it is dropped (ADR-0067).
+- **`restore()` reads live rows only.** The store answers `WHERE status = 'live'`; terminal rows are not parsed at boot.
+- **A finished run's row lives 7 days, fixed.** Every non-live status — `done`, `failed`, `error`, `cancelled`, `lost`,
+  `drifted` — ages from its terminal write, and a sweep at boot and every hour deletes what is older. A `live` row is
+  never swept: a run parked on a Gate for a month stays and is restored. The value is fixed like the Sandbox idle
+  timeout, not config: jr2 is not the record of a run's results (a Machine sends its output where it belongs — a PR, a
+  Source), and `jr2 status` on an old run is a debugging act. A swept id frees its abbreviations (ADR-0009).
+- **The store is sqlite on the Instance's volume. No Postgres is owed.** The `SnapshotStore` interface stays as the
+  seam, with one implementation. `DATABASE_URL` is not a thing jr2 reads.
+
+## Considered options
+
+- **Keep `synchronous=FULL`, move the write to a worker thread.** Rejected: the thread hides 4.5 ms from the loop but
+  does not make the disk faster, so writes queue under load and the loop no longer knows how far behind the store is —
+  and it opens the ordering gap on every write, which NORMAL opens only on a power loss.
+- **Batch persists in a window.** Rejected: one persist per macrostep already coalesces a transition's events, and with
+  the fsync gone there is nothing left to amortize.
+- **A row per worker.** Rejected: it splits an atomic tree across rows (a worker's row saved, the Pool's not), makes
+  restore reassemble what xstate restores whole, and buys nothing measurable at 0.2 ms per blob.
+- **Configurable retention (`runs.retain`), or a count cap per workflow.** Rejected: a count cap does not bound a busy
+  workflow's disk and punishes a quiet one; a config key is owed only when a history surface (a list, a query) exists to
+  use it, and none does — `jr2 runs` and the Console list live runs.
+- **Build the Postgres store.** Rejected: the only thing it buys is an Orchestrator pod that reschedules without its
+  volume, which is a storage-class question (a network RWO volume follows the pod) and belongs to the Orchestrator pod's
+  own decision (R10), not to the store. Its cost is a dependency the owner operates, a second store in every test tier,
+  and the asynchronous write above.
+
+## Consequences
+
+- ADR-0008 and ADR-0009 lose "Postgres opt-in" and `DATABASE_URL`; ADR-0008's Orchestrator persists its snapshots "to
+  sqlite on its volume". ADR-0007 points here for the store's durability.
+- The Instance's volume gains `state.db-wal` and `state.db-shm` beside `state.db`.
+- A `drifted` run (ADR-0030) is swept a week after the deploy that changed its Machine, not a week after it ended; a
+  week is the time a human has to read it.