@jr2/orchestrator 0.4.1 → 0.5.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (327) hide show
  1. package/README.md +20 -2
  2. package/console/components/app.ts +2 -0
  3. package/console/components/fleet.ts +14 -1
  4. package/console/components/machine-pane.ts +41 -2
  5. package/console/main.ts +13 -1
  6. package/console/store.ts +102 -6
  7. package/console/style.css +21 -0
  8. package/docs/agents-block.md +5 -0
  9. package/docs/concepts/agents-and-turns.md +442 -0
  10. package/docs/concepts/architecture.md +230 -0
  11. package/docs/concepts/charts.md +143 -0
  12. package/docs/concepts/composition.md +357 -0
  13. package/docs/concepts/converge.md +254 -0
  14. package/docs/concepts/durable-runs.md +189 -0
  15. package/docs/concepts/effects.md +603 -0
  16. package/docs/concepts/gates.md +208 -0
  17. package/docs/concepts/machines-and-workflows.md +289 -0
  18. package/docs/concepts/pools-and-sources.md +230 -0
  19. package/docs/concepts/secrets-and-trust.md +222 -0
  20. package/docs/concepts/the-menu.md +378 -0
  21. package/docs/concepts/workspaces.md +678 -0
  22. package/docs/get-started/a-model-decides.md +563 -0
  23. package/docs/get-started/an-agent-in-a-sandbox.md +381 -0
  24. package/docs/get-started/connect-a-model.md +248 -0
  25. package/docs/get-started/create-an-instance.md +194 -0
  26. package/docs/get-started/have-your-assistant-write-a-machine.md +922 -0
  27. package/docs/get-started/install.md +126 -0
  28. package/docs/guides/add-tools-to-the-sandbox.md +295 -0
  29. package/docs/guides/answer-a-gate.md +304 -0
  30. package/docs/guides/bind-open-parts.md +223 -0
  31. package/docs/guides/cancel-a-run.md +146 -0
  32. package/docs/guides/clean-up.md +206 -0
  33. package/docs/guides/fix-a-failed-converge.md +361 -0
  34. package/docs/guides/give-the-cluster-git-credentials.md +222 -0
  35. package/docs/guides/iterate-on-a-machine-on-kind.md +190 -0
  36. package/docs/guides/let-an-agent-write-a-chart.md +226 -0
  37. package/docs/guides/look-inside-a-sandbox.md +174 -0
  38. package/docs/guides/mirror-the-kit-images.md +176 -0
  39. package/docs/guides/move-from-kind-to-a-shared-cluster.md +302 -0
  40. package/docs/guides/open-the-console.md +91 -0
  41. package/docs/guides/publish-a-machine-as-an-npm-package.md +350 -0
  42. package/docs/guides/push-and-sign-a-branch.md +486 -0
  43. package/docs/guides/run-a-chart-by-name.md +270 -0
  44. package/docs/guides/run-a-command-or-call-a-url.md +544 -0
  45. package/docs/guides/run-one-worker-per-item.md +321 -0
  46. package/docs/guides/set-an-agents-access-and-dials.md +375 -0
  47. package/docs/guides/size-a-workspace.md +204 -0
  48. package/docs/guides/start-a-run-from-a-run.md +237 -0
  49. package/docs/guides/start-a-run.md +207 -0
  50. package/docs/guides/upgrade-the-kit-version.md +240 -0
  51. package/docs/guides/use-a-hosted-provider.md +208 -0
  52. package/docs/guides/use-a-local-model.md +313 -0
  53. package/docs/guides/watch-a-run.md +326 -0
  54. package/docs/guides/write-a-workflow.md +318 -0
  55. package/docs/index.md +44 -0
  56. package/docs/llms.txt +216 -0
  57. package/docs/reference/authoring-surface/agent.md +128 -0
  58. package/docs/reference/authoring-surface/answer.md +140 -0
  59. package/docs/reference/authoring-surface/arg.md +60 -0
  60. package/docs/reference/authoring-surface/basic.md +62 -0
  61. package/docs/reference/authoring-surface/bearer.md +53 -0
  62. package/docs/reference/authoring-surface/chartschema.md +23 -0
  63. package/docs/reference/authoring-surface/chartschemafor.md +48 -0
  64. package/docs/reference/authoring-surface/command.md +191 -0
  65. package/docs/reference/authoring-surface/customize.md +64 -0
  66. package/docs/reference/authoring-surface/defineconfig.md +41 -0
  67. package/docs/reference/authoring-surface/defineevent.md +114 -0
  68. package/docs/reference/authoring-surface/dispatch.md +126 -0
  69. package/docs/reference/authoring-surface/fail.md +83 -0
  70. package/docs/reference/authoring-surface/file.md +122 -0
  71. package/docs/reference/authoring-surface/frompick.md +277 -0
  72. package/docs/reference/authoring-surface/http.md +159 -0
  73. package/docs/reference/authoring-surface/isfailure.md +58 -0
  74. package/docs/reference/authoring-surface/jr2setup.md +237 -0
  75. package/docs/reference/authoring-surface/open.md +74 -0
  76. package/docs/reference/authoring-surface/openbody.md +86 -0
  77. package/docs/reference/authoring-surface/paramsof.md +125 -0
  78. package/docs/reference/authoring-surface/pool.md +186 -0
  79. package/docs/reference/authoring-surface/push.md +112 -0
  80. package/docs/reference/authoring-surface/secret.md +118 -0
  81. package/docs/reference/authoring-surface/sentby.md +83 -0
  82. package/docs/reference/authoring-surface/source.md +51 -0
  83. package/docs/reference/authoring-surface/text.md +62 -0
  84. package/docs/reference/authoring-surface/type-agentdeclaration.md +27 -0
  85. package/docs/reference/authoring-surface/type-agentdefinition.md +80 -0
  86. package/docs/reference/authoring-surface/type-agentlogic.md +15 -0
  87. package/docs/reference/authoring-surface/type-agentslots.md +12 -0
  88. package/docs/reference/authoring-surface/type-answeraction.md +12 -0
  89. package/docs/reference/authoring-surface/type-arg.md +19 -0
  90. package/docs/reference/authoring-surface/type-argnames.md +12 -0
  91. package/docs/reference/authoring-surface/type-binding.md +62 -0
  92. package/docs/reference/authoring-surface/type-bindingof.md +12 -0
  93. package/docs/reference/authoring-surface/type-bodyacceptsdoor.md +15 -0
  94. package/docs/reference/authoring-surface/type-bodyacceptsslots.md +17 -0
  95. package/docs/reference/authoring-surface/type-ceilings.md +41 -0
  96. package/docs/reference/authoring-surface/type-chainlink.md +19 -0
  97. package/docs/reference/authoring-surface/type-chartaction.md +46 -0
  98. package/docs/reference/authoring-surface/type-chartagent.md +28 -0
  99. package/docs/reference/authoring-surface/type-chartevent.md +20 -0
  100. package/docs/reference/authoring-surface/type-chartguard.md +20 -0
  101. package/docs/reference/authoring-surface/type-chartinvoke.md +28 -0
  102. package/docs/reference/authoring-surface/type-chartjson.md +14 -0
  103. package/docs/reference/authoring-surface/type-chartstate.md +46 -0
  104. package/docs/reference/authoring-surface/type-charttext.md +40 -0
  105. package/docs/reference/authoring-surface/type-charttransition.md +22 -0
  106. package/docs/reference/authoring-surface/type-charttransitions.md +14 -0
  107. package/docs/reference/authoring-surface/type-chartvalue.md +15 -0
  108. package/docs/reference/authoring-surface/type-childat.md +12 -0
  109. package/docs/reference/authoring-surface/type-childslots.md +12 -0
  110. package/docs/reference/authoring-surface/type-commandenv.md +15 -0
  111. package/docs/reference/authoring-surface/type-commandinput.md +19 -0
  112. package/docs/reference/authoring-surface/type-commandoptions.md +78 -0
  113. package/docs/reference/authoring-surface/type-commandoutput.md +104 -0
  114. package/docs/reference/authoring-surface/type-customize.md +109 -0
  115. package/docs/reference/authoring-surface/type-dispatchdefinition.md +39 -0
  116. package/docs/reference/authoring-surface/type-dispatchinput.md +14 -0
  117. package/docs/reference/authoring-surface/type-dispatchopenparts.md +14 -0
  118. package/docs/reference/authoring-surface/type-dispatchoutput.md +65 -0
  119. package/docs/reference/authoring-surface/type-duration.md +15 -0
  120. package/docs/reference/authoring-surface/type-effectbinding.md +93 -0
  121. package/docs/reference/authoring-surface/type-effectlogic.md +12 -0
  122. package/docs/reference/authoring-surface/type-envopenparts.md +14 -0
  123. package/docs/reference/authoring-surface/type-eventaudience.md +27 -0
  124. package/docs/reference/authoring-surface/type-eventdef.md +45 -0
  125. package/docs/reference/authoring-surface/type-eventfrom.md +16 -0
  126. package/docs/reference/authoring-surface/type-eventsemantics.md +26 -0
  127. package/docs/reference/authoring-surface/type-execoutcome.md +34 -0
  128. package/docs/reference/authoring-surface/type-failslot.md +14 -0
  129. package/docs/reference/authoring-surface/type-failure.md +19 -0
  130. package/docs/reference/authoring-surface/type-failureinput.md +20 -0
  131. package/docs/reference/authoring-surface/type-filebody.md +41 -0
  132. package/docs/reference/authoring-surface/type-frominput.md +14 -0
  133. package/docs/reference/authoring-surface/type-hostinjectedinput.md +58 -0
  134. package/docs/reference/authoring-surface/type-httpdefinition.md +101 -0
  135. package/docs/reference/authoring-surface/type-httpinput.md +12 -0
  136. package/docs/reference/authoring-surface/type-httpmethod.md +14 -0
  137. package/docs/reference/authoring-surface/type-httpopenparts.md +12 -0
  138. package/docs/reference/authoring-surface/type-httpoutput.md +89 -0
  139. package/docs/reference/authoring-surface/type-httprequestparts.md +33 -0
  140. package/docs/reference/authoring-surface/type-jr2config.md +126 -0
  141. package/docs/reference/authoring-surface/type-lends.md +38 -0
  142. package/docs/reference/authoring-surface/type-logicat.md +12 -0
  143. package/docs/reference/authoring-surface/type-markedif.md +12 -0
  144. package/docs/reference/authoring-surface/type-mergedactors.md +12 -0
  145. package/docs/reference/authoring-surface/type-namedsecret.md +25 -0
  146. package/docs/reference/authoring-surface/type-noslotspec.md +26 -0
  147. package/docs/reference/authoring-surface/type-noslotworkspaceoptions.md +32 -0
  148. package/docs/reference/authoring-surface/type-openbodydeclaration.md +40 -0
  149. package/docs/reference/authoring-surface/type-openbodylogic.md +14 -0
  150. package/docs/reference/authoring-surface/type-openbodyoutput.md +42 -0
  151. package/docs/reference/authoring-surface/type-openbodyslot.md +18 -0
  152. package/docs/reference/authoring-surface/type-opencommandinput.md +17 -0
  153. package/docs/reference/authoring-surface/type-openeffect.md +12 -0
  154. package/docs/reference/authoring-surface/type-openeffectslots.md +12 -0
  155. package/docs/reference/authoring-surface/type-openof.md +12 -0
  156. package/docs/reference/authoring-surface/type-opensecret.md +23 -0
  157. package/docs/reference/authoring-surface/type-opensecretmark.md +12 -0
  158. package/docs/reference/authoring-surface/type-permissivenoslotworkspaceoptions.md +31 -0
  159. package/docs/reference/authoring-surface/type-permissiveworkspaceoptions.md +32 -0
  160. package/docs/reference/authoring-surface/type-poolmachine.md +28 -0
  161. package/docs/reference/authoring-surface/type-poolspec.md +60 -0
  162. package/docs/reference/authoring-surface/type-pushinput.md +15 -0
  163. package/docs/reference/authoring-surface/type-pushoptions.md +79 -0
  164. package/docs/reference/authoring-surface/type-pushoutput.md +124 -0
  165. package/docs/reference/authoring-surface/type-pushwireoutput.md +12 -0
  166. package/docs/reference/authoring-surface/type-quantity.md +14 -0
  167. package/docs/reference/authoring-surface/type-reached.md +12 -0
  168. package/docs/reference/authoring-surface/type-reposlot.md +70 -0
  169. package/docs/reference/authoring-surface/type-reposlotsof.md +16 -0
  170. package/docs/reference/authoring-surface/type-sandboxoptions.md +77 -0
  171. package/docs/reference/authoring-surface/type-secret.md +25 -0
  172. package/docs/reference/authoring-surface/type-sender.md +19 -0
  173. package/docs/reference/authoring-surface/type-sentbyguards.md +12 -0
  174. package/docs/reference/authoring-surface/type-size.md +85 -0
  175. package/docs/reference/authoring-surface/type-slothandles.md +32 -0
  176. package/docs/reference/authoring-surface/type-sourcenextresult.md +17 -0
  177. package/docs/reference/authoring-surface/type-sourcespec.md +44 -0
  178. package/docs/reference/authoring-surface/type-syncstate.md +12 -0
  179. package/docs/reference/authoring-surface/type-text.md +19 -0
  180. package/docs/reference/authoring-surface/type-textpart.md +24 -0
  181. package/docs/reference/authoring-surface/type-thinkinglevel.md +39 -0
  182. package/docs/reference/authoring-surface/type-toparameterizedobject.md +14 -0
  183. package/docs/reference/authoring-surface/type-toprovidedactor.md +14 -0
  184. package/docs/reference/authoring-surface/type-unknown.md +21 -0
  185. package/docs/reference/authoring-surface/type-usercontainer.md +77 -0
  186. package/docs/reference/authoring-surface/type-withoutemptyindex.md +20 -0
  187. package/docs/reference/authoring-surface/type-workflowevent.md +46 -0
  188. package/docs/reference/authoring-surface/type-workspaceaccess.md +16 -0
  189. package/docs/reference/authoring-surface/type-workspaced.md +94 -0
  190. package/docs/reference/authoring-surface/type-workspacehandles.md +23 -0
  191. package/docs/reference/authoring-surface/type-workspacemachine.md +29 -0
  192. package/docs/reference/authoring-surface/type-workspaceof.md +13 -0
  193. package/docs/reference/authoring-surface/type-workspaceoptions.md +39 -0
  194. package/docs/reference/authoring-surface/type-workspacespec.md +37 -0
  195. package/docs/reference/authoring-surface/workspace.md +259 -0
  196. package/docs/reference/authoring-surface.md +197 -0
  197. package/docs/reference/cli.md +433 -0
  198. package/docs/reference/config.md +756 -0
  199. package/docs/reference/decisions/0001-operator-owned-generic-sandbox.md +113 -0
  200. package/docs/reference/decisions/0002-duplex-streaming-actor.md +110 -0
  201. package/docs/reference/decisions/0004-worktree-provisioning-model.md +124 -0
  202. package/docs/reference/decisions/0005-sandbox-pod-composition.md +230 -0
  203. package/docs/reference/decisions/0006-agent-control-surface.md +47 -0
  204. package/docs/reference/decisions/0007-durable-machine-state.md +109 -0
  205. package/docs/reference/decisions/0008-orchestrator-as-deployed-app.md +60 -0
  206. package/docs/reference/decisions/0009-cli-and-instance-interface.md +275 -0
  207. package/docs/reference/decisions/0010-bdd-acceptance-tests.md +160 -0
  208. package/docs/reference/decisions/0011-workflow-defined-events.md +141 -0
  209. package/docs/reference/decisions/0012-workspace-wrapper-machine.md +103 -0
  210. package/docs/reference/decisions/0013-the-agent-reaches-its-machine-through-a-container-it-cannot-read.md +157 -0
  211. package/docs/reference/decisions/0014-observation-is-open-run-state-is-not.md +95 -0
  212. package/docs/reference/decisions/0015-authoring-surface-absorbs-the-mechanism.md +265 -0
  213. package/docs/reference/decisions/0016-agent-turn-mechanics-are-internal.md +124 -0
  214. package/docs/reference/decisions/0017-pool-over-a-source-port.md +69 -0
  215. package/docs/reference/decisions/0018-instance-agents-are-definitions-jr2-assembles-the-harness.md +111 -0
  216. package/docs/reference/decisions/0019-one-converging-command-against-the-current-context.md +275 -0
  217. package/docs/reference/decisions/0020-private-ca-trust-is-a-named-concept.md +60 -0
  218. package/docs/reference/decisions/0021-workspace-continuity-is-a-lease-that-answers-back.md +100 -0
  219. package/docs/reference/decisions/0022-observation-is-a-level-triggered-feed.md +103 -0
  220. package/docs/reference/decisions/0023-the-harness-prints-the-conversation.md +176 -0
  221. package/docs/reference/decisions/0024-an-agents-turn-ends-with-the-state-that-asked-for-it.md +439 -0
  222. package/docs/reference/decisions/0025-cancel-ends-a-run-stop-parks-it.md +65 -0
  223. package/docs/reference/decisions/0026-a-turn-that-is-over-is-never-prompted.md +62 -0
  224. package/docs/reference/decisions/0027-the-harness-is-jr2s-own-server-flue-retires-the-wire-stays.md +237 -0
  225. package/docs/reference/decisions/0028-what-an-agent-may-do-to-the-workspace-is-part-of-its-definition.md +89 -0
  226. package/docs/reference/decisions/0029-a-menu-is-fixed-for-a-conversation-and-a-turn-is-told-its-allowed-picks.md +154 -0
  227. package/docs/reference/decisions/0030-a-snapshot-names-the-machine-it-was-written-under.md +100 -0
  228. package/docs/reference/decisions/0031-menu-only-agents-run-on-the-instance-harness.md +190 -0
  229. package/docs/reference/decisions/0032-the-console-unlocks-with-the-instance-token.md +61 -0
  230. package/docs/reference/decisions/0033-a-machine-declares-the-input-that-starts-it.md +127 -0
  231. package/docs/reference/decisions/0034-the-console-renders-with-preact-and-the-server-erases-the-types.md +57 -0
  232. package/docs/reference/decisions/0035-a-runaway-turn-is-ended-rerolled-once-then-a-fault.md +92 -0
  233. package/docs/reference/decisions/0036-a-turn-compacts-its-own-context-mid-flight.md +138 -0
  234. package/docs/reference/decisions/0037-an-instance-builds-its-sandbox-images-jr2-injects-the-harness.md +196 -0
  235. package/docs/reference/decisions/0038-jr2-up-builds-every-image-it-deploys.md +170 -0
  236. package/docs/reference/decisions/0039-image-garbage-collects-by-reachability.md +115 -0
  237. package/docs/reference/decisions/0041-a-build-the-host-already-holds-is-not-spent-again.md +64 -0
  238. package/docs/reference/decisions/0042-ready-is-not-routable-an-admission-retries-its-connection.md +142 -0
  239. package/docs/reference/decisions/0043-the-kit-is-tested-as-installed-a-local-registry-stands-in-for-npm.md +170 -0
  240. package/docs/reference/decisions/0044-kit-images-live-at-a-canonical-home-a-self-host-mirrors-it.md +85 -0
  241. package/docs/reference/decisions/0045-the-platform-joins-the-image-address-and-the-cluster-chooses-it.md +82 -0
  242. package/docs/reference/decisions/0046-a-converge-failure-carries-its-diagnosis.md +34 -0
  243. package/docs/reference/decisions/0047-the-git-ssh-key-source-is-the-users-choice.md +47 -0
  244. package/docs/reference/decisions/0048-the-orchestrator-boots-without-its-repos.md +36 -0
  245. package/docs/reference/decisions/0049-a-machine-carries-its-parts-and-composes-by-invoke.md +126 -0
  246. package/docs/reference/decisions/0050-what-code-names-config-declares-what-the-cli-names-discovery-finds.md +73 -0
  247. package/docs/reference/decisions/0051-a-repo-is-a-slot-on-the-workspace-and-a-cache-on-the-node.md +227 -0
  248. package/docs/reference/decisions/0052-a-sandbox-lands-where-an-ordinary-pod-lands-and-the-cache-follows-it.md +81 -0
  249. package/docs/reference/decisions/0053-a-fetch-inside-the-pod-asks-the-node-cache-and-the-cache-asks-the-remote.md +111 -0
  250. package/docs/reference/decisions/0054-the-kit-ships-machines-and-a-shipped-machine-leaves-its-parts-open.md +126 -0
  251. package/docs/reference/decisions/0055-a-release-is-a-pushed-tag-and-the-publish-guard-is-the-credential.md +107 -0
  252. package/docs/reference/decisions/0056-a-global-jr2-hands-off-to-the-instances-own-and-an-instance-has-one-kit-version.md +62 -0
  253. package/docs/reference/decisions/0057-a-turn-is-its-frame-its-dials-and-whether-it-continues.md +152 -0
  254. package/docs/reference/decisions/0058-a-harness-answers-the-orchestrator-alone.md +106 -0
  255. package/docs/reference/decisions/0059-a-harness-holds-stand-ins-and-the-custodian-holds-the-keys.md +384 -0
  256. package/docs/reference/decisions/0060-a-workspace-states-its-size-and-jr2-reserves-all-of-it.md +111 -0
  257. package/docs/reference/decisions/0061-a-memory-kill-ends-the-agents-processes-not-the-harness.md +83 -0
  258. package/docs/reference/decisions/0062-jr2-briefs-the-agent-on-its-seat-never-on-its-task.md +212 -0
  259. package/docs/reference/decisions/0063-the-orchestrator-watches-the-cluster.md +83 -0
  260. package/docs/reference/decisions/0064-a-workspace-waits-for-capacity.md +79 -0
  261. package/docs/reference/decisions/0065-a-persist-survives-a-crash-and-history-is-a-week.md +65 -0
  262. package/docs/reference/decisions/0066-a-model-tier-runs-a-real-model-and-a-change-to-what-it-reads-runs-the-tier.md +550 -0
  263. package/docs/reference/decisions/0067-an-effect-is-an-actor-slot-and-runs-at-most-once-unless-marked-idempotent.md +430 -0
  264. package/docs/reference/decisions/0068-an-effector-holds-an-effects-secrets-and-signs-its-wire.md +464 -0
  265. package/docs/reference/decisions/0069-the-orchestrator-is-a-kit-image-and-an-instance-publishes-its-assembly.md +180 -0
  266. package/docs/reference/decisions/0070-a-machine-lends-its-parts-to-a-chart.md +323 -0
  267. package/docs/reference/decisions/0071-a-run-starts-a-run-through-a-dispatch-effect.md +183 -0
  268. package/docs/reference/decisions/0072-the-docs-ship-inside-the-kit-and-an-instances-agents-md-indexes-them.md +144 -0
  269. package/docs/reference/decisions/0073-a-machine-fails-a-run-with-fail-and-an-uncaught-failure-ends-it-failed.md +122 -0
  270. package/docs/reference/decisions/0074-an-unrouted-mechanism-event-is-refused-at-load-and-faults-the-run.md +76 -0
  271. package/docs/reference/decisions.md +83 -0
  272. package/docs/reference/glossary.md +648 -0
  273. package/docs/reference/http-api.md +1348 -0
  274. package/docs/reference/machines.md +190 -0
  275. package/docs/reference/mechanism-events.md +72 -0
  276. package/docs/why.md +206 -0
  277. package/package.json +5 -3
  278. package/src/actor.ts +541 -79
  279. package/src/agent.ts +114 -38
  280. package/src/ambient.ts +2 -1
  281. package/src/assembly-checks.ts +189 -7
  282. package/src/cause.ts +234 -0
  283. package/src/chart.ts +169 -37
  284. package/src/charts.ts +34 -6
  285. package/src/config.ts +430 -178
  286. package/src/custodian.ts +74 -22
  287. package/src/customize.ts +188 -46
  288. package/src/dispatch-effect.ts +945 -0
  289. package/src/door-compat.ts +217 -0
  290. package/src/durability.ts +11 -4
  291. package/src/effect.ts +341 -44
  292. package/src/effector-wire.ts +160 -11
  293. package/src/fail.ts +209 -0
  294. package/src/harness-client.ts +541 -47
  295. package/src/http-effect.ts +495 -53
  296. package/src/http.ts +427 -16
  297. package/src/index.ts +60 -82
  298. package/src/instance.ts +68 -8
  299. package/src/internal.ts +84 -0
  300. package/src/machine-doc.ts +21 -3
  301. package/src/mechanism-routes.ts +168 -0
  302. package/src/model-keys.generated.ts +53 -0
  303. package/src/model-keys.ts +18 -58
  304. package/src/notices.ts +66 -13
  305. package/src/open-body.ts +159 -23
  306. package/src/open-effect.ts +81 -8
  307. package/src/open.ts +59 -6
  308. package/src/parts.ts +184 -13
  309. package/src/party.ts +68 -2
  310. package/src/pick.ts +463 -11
  311. package/src/pool.ts +247 -36
  312. package/src/registration.ts +83 -4
  313. package/src/repos.ts +2 -1
  314. package/src/run-host.ts +565 -31
  315. package/src/run-output.ts +38 -0
  316. package/src/sandbox-effect.ts +273 -38
  317. package/src/sandbox-kube.ts +3 -1
  318. package/src/secret-placement.ts +52 -17
  319. package/src/server.ts +3 -0
  320. package/src/setup.ts +239 -56
  321. package/src/size.ts +121 -9
  322. package/src/snapshot-store.ts +54 -10
  323. package/src/stub-harness.ts +44 -16
  324. package/src/timers.ts +14 -1
  325. package/src/vocabulary.ts +69 -14
  326. package/src/wire.ts +90 -16
  327. package/src/workspace.ts +697 -175
@@ -0,0 +1,384 @@
1
+ # A Harness holds stand-ins; the Custodian holds the keys
2
+
3
+ [ADR-0058](0058-a-harness-answers-the-orchestrator-alone.md) recorded an open gap: model API keys were in the Harness
4
+ container. `JR2_PROVIDER_API_KEY` and literal `harness.env` values (for example `ANTHROPIC_API_KEY`) came from the
5
+ `jr2-harness-env` Secret by `envFrom`. Working tools give the Agent code execution in that container, and every tool
6
+ child inherits the env. So the Agent could read a key and send it out of the pod. Dropping pi's catalog was not an
7
+ option: the user's real setup is Bedrock through a LiteLLM gateway, with a fixed `ANTHROPIC_API_KEY` and `anthropic/…`
8
+ models, and the catalog holds the limits, costs and thinking levels of those models.
9
+
10
+ The same problem had one solution already, for one credential. The Adapter held the Sandbox token in a container the
11
+ Agent executes nothing in ([ADR-0013](0013-the-agent-reaches-its-machine-through-a-container-it-cannot-read.md)). This
12
+ decision generalizes that container. It holds every credential the Harness in its pod uses, and it replaces the Adapter.
13
+
14
+ ## Decision
15
+
16
+ - **A held secret is a credential the Harness uses and never holds.** The Harness container's env holds its
17
+ **Stand-in**, `jr2-held-<NAME>`. The Stand-in is deterministic and public. It means something only to the Custodian,
18
+ and only toward a host the secret is bound to. It is unique per name, it is a valid header and env value, and it never
19
+ contains `sk-ant-oat` (pi reads such a key as an Anthropic OAuth token). The word is not "placeholder", because that
20
+ is the word of an Open part.
21
+ - **The Custodian holds them.** It is a container in every Harness pod — every Sandbox and the Instance Harness — beside
22
+ the Harness, and it holds the held secrets its own pod names (below). The Agent executes no code in it, and
23
+ `shareProcessNamespace` stays off ([ADR-0005](0005-sandbox-pod-composition.md)). So the Agent cannot read its env, its
24
+ files or its `/proc`. Its engine is Envoy: the upstream distroless image, pinned by digest (`CUSTODIAN_IMAGE` in
25
+ `@jr2/orchestrator`). The pod's maker writes Envoy's whole configuration — the Orchestrator a Sandbox's at provision,
26
+ `jr2 up` the Instance Harness's — from the manifest `jr2 up` declares (see "Objects, per pod"). There is no admin
27
+ listener and no xDS, so nothing in the pod can change the configuration or read it back.
28
+ - **The Custodian replaces the Adapter.** The `@jr2/adapter` package and the `jr2-adapter` image are deleted. The
29
+ Sandbox token is a held secret of the kit's own: the Harness container holds
30
+ `JR2_SANDBOX_TOKEN=jr2-held-JR2_SANDBOX_TOKEN`, and the Custodian holds the token as a file from the `<sandbox>-token`
31
+ Secret (the Instance Harness: the `JR2_INSTANCE_HARNESS_TOKEN` key of `jr2-instance`). No other container mounts it.
32
+ - **The Harness reads its Menu itself.** There is no MCP. For each Submission, `menu.ts` reads
33
+ `GET /agents/<iid>/surface` and delivers a pick with `POST /agents/<iid>/events`, both to the Custodian on
34
+ `127.0.0.1:8081`, with the Stand-in as the bearer. `menu-tools.ts` gives pi each accepted event as a tool. The tool
35
+ names stay `mcp__jr2__<event>`, because shipped instructions and the printer depend on them; the name is no longer a
36
+ transport. `menu-tools.ts` adds the prefix, and `menu.ts` keeps the bare name, because an MCP client adds
37
+ `mcp__<server>__` itself. The Harness keeps what the Adapter did: a turn that is over is never prompted
38
+ ([ADR-0026](0026-a-turn-that-is-over-is-never-prompted.md)), a receipt is prose
39
+ ([ADR-0024](0024-an-agents-turn-ends-with-the-state-that-asked-for-it.md),
40
+ [ADR-0029](0029-a-menu-is-fixed-for-a-conversation-and-a-turn-is-told-its-allowed-picks.md)), the surface read retries
41
+ on the ladder with the `jr2.routability` line
42
+ ([ADR-0042](0042-ready-is-not-routable-an-admission-retries-its-connection.md)), and a `poll` event refuses the turn
43
+ (`deferred` is retired, ADR-0006). The event's own JSON Schema is the tool's parameters, so the model sees the
44
+ contract the Orchestrator validates against.
45
+ - **The control listener forwards four routes and no other.** `127.0.0.1:8081`, plain HTTP, to the Orchestrator's
46
+ Service (`jr2-orchestrator.<ns>.svc:4000`, written into the pod's bootstrap by whoever makes the pod — see "Objects,
47
+ per pod"):
48
+
49
+ ```
50
+ GET /agents/<iid>/surface → the same path the Menu
51
+ POST /agents/<iid>/events → the same path a pick
52
+ GET /agents/<iid>/deliveries/<id> → the same path an outstanding pick's answer (ADR-0024)
53
+ POST /fetch → /sandboxes/<this pod's Sandbox>/fetch the ask (ADR-0053)
54
+ anything else → 404, from the Custodian
55
+ ```
56
+
57
+ A read — the surface, a delivery's poll — is retried on any reset; a pick and the ask only on a reset before they were
58
+ sent, so one the Orchestrator may have read is not sent twice. A poll holds up to about 25 s, far below the listener's
59
+ 3600 s idle limit, so an outstanding pick never rides one long request (ADR-0002).
60
+
61
+ On each forwarded request, the Custodian swaps `Authorization: Bearer <Stand-in>` for the token and removes every
62
+ other credential header. A request without the Stand-in gets 403. The ask's path comes from the Custodian
63
+ (`JR2_SANDBOX` in its env, set per pod); on the Instance Harness there is none, and the ask gets 404. The route
64
+ allowlist is the second layer. The control stays in the Orchestrator: the token's scope, the live registration, and
65
+ the event's name and payload are judged there, as before (ADR-0013).
66
+
67
+ - **Model keys go the same way, over HTTPS.** When its pod holds a held secret, the Harness container also gets:
68
+
69
+ ```sh
70
+ <NAME>=jr2-held-<NAME> one per held secret of the pod
71
+ HTTPS_PROXY / https_proxy=http://127.0.0.1:15001
72
+ NO_PROXY / no_proxy=localhost,127.0.0.1,::1
73
+ NODE_USE_ENV_PROXY=1
74
+ NODE_EXTRA_CA_CERTS=/etc/jr2/ca/extra.crt
75
+ SSL_CERT_FILE / REQUESTS_CA_BUNDLE / GIT_SSL_CAINFO=/etc/jr2/ca/bundle.crt
76
+ ```
77
+
78
+ These ride `env`, after `harness.env` and before the wire's digest. In Kubernetes `env` wins over `envFrom`, so a
79
+ Stand-in overrides a same-named key of an `envFrom` Secret, even one that changes later. There is no `HTTP_PROXY`: the
80
+ Custodian carries TLS only, and plain HTTP (the Custodian's control listener, an in-cluster seed repository) goes
81
+ direct. `NO_PROXY` is loopback only, so an in-cluster gateway that a secret is bound to still goes through the
82
+ Custodian.
83
+
84
+ - **The egress listener: CONNECT only.** `127.0.0.1:15001`. The CONNECT target decides everything (lowercased, without a
85
+ trailing dot) — never the SNI, never the `Host` header. A target that a held secret is bound to goes to an internal
86
+ listener for that target alone. That listener ends TLS with a leaf `jr2 up` issued (ALPN `http/1.1` only) and, per
87
+ request:
88
+ - The method must be GET, HEAD, POST, PUT, PATCH, DELETE or OPTIONS, else 405. A server that answers TRACE echoes the
89
+ request, the value among it, back to the Agent. A header name with `_` gives 400: `x_api_key` is not `x-api-key` to
90
+ the strip, but some servers read it as one.
91
+ - `Host` must name the target, else 421. An encoded `/`, `.` or `\` in the path gives 400. A path outside the secret's
92
+ `paths` gives 403. It is checked after Envoy removes dot segments and reads `\` as `/` and `..;` as `..`, and the
93
+ upstream gets that path. So IIS and Tomcat cannot read a different path. A credential header sent twice gives 400.
94
+ - The swap: in each header the secret names, the Stand-in alone, or one auth scheme and the Stand-in
95
+ (`Bearer <Stand-in>`), becomes the value. Nothing else is rewritten.
96
+ - Every credential header that is not the product of a swap is removed. A request with no swap gets 403 and never
97
+ leaves the pod: `jr2 custodian: <target> needs the stand-in of <NAME> in <headers>`.
98
+ - Envoy opens its own TLS to the target, verifies the chain against `upstream.crt` and the name against the target's
99
+ SAN (DNS or IP), and adds no header either way (`server`, `x-envoy-*`, `x-request-id` and `x-forwarded-*` are off).
100
+ A failure gives 502 and a body that names the target and the reason. Redirects pass back unchanged.
101
+ - When the target gives no answer, the Custodian answers 502 with a body that names the target, says what jr2 knows
102
+ about the request, and ends with Envoy's reason. It replaces Envoy's own answer for ten Envoy response flags, in two
103
+ mappers. One reply can carry several flags, and Envoy uses the first mapper that matches, in order. The first mapper
104
+ is for `UF`, `UH` and `DF`, so a reply with one of them gets the first text whatever else it carries:
105
+ - `UF` (the connection to the target failed), `UH` (no healthy host) and `DF` (the DNS lookup failed) say that Envoy
106
+ did not send the request. The body is `jr2 custodian: <target> could not be reached: <details> <reason>`.
107
+ - `UC` (the target closed the connection), `DPE` (a downstream protocol error, which a body framing error can cause
108
+ after the headers went), `UPE` (an upstream protocol error), `UR` (the target reset the stream) and `LR` (Envoy
109
+ reset it) can come after the request went to the target. The body is
110
+ `jr2 custodian: <target> did not answer; the request may have reached it: <details> <reason>`. `UMSDR` (the
111
+ maximum stream duration) is in this mapper: it ends a stream that was sent. `URX` (the retry limit) is in it too:
112
+ Envoy sets it with the last attempt's own flag, so `UF,URX` gets the first text and `UC,URX` the second.
113
+ - Envoy does not set `URX` or `UMSDR` on these routes, because they set no retry policy and no
114
+ `max_stream_duration`. The two flags are in the mappers only to keep their union equal to the Orchestrator route's
115
+ list.
116
+
117
+ The second text is there because the first would be false after those flags. The target may have the request, so a
118
+ `curl -X POST` or a `gh api` call to a bound host can write twice when the Agent sends it again. "Could not be
119
+ reached" would tell the Agent that the first call went nowhere. The status stays 502 for both, the status of a
120
+ gateway whose target gave no valid answer. No code reads the words. The Harness's provider adapter reads the status,
121
+ and no model reads its own provider's 502. The Agent's own tools show the body to the model.
122
+
123
+ Without the mappers, Envoy answers each of the ten flags itself: 503 for most of them, and 502 for `UPE`. That body
124
+ names no target, so a caller can read it as the target's own answer. With them, every caller sees that the Custodian
125
+ gave the answer, and for which target. A reset after the target's answer started is not replaced: the caller sees
126
+ its stream end.
127
+
128
+ The tunnel (below) keeps one mapper for all ten flags, with the first text. Envoy answers a `CONNECT` with 200 only
129
+ after its TCP connection to the target is up, and no byte of the Agent's request goes before that 200. So a 502 on
130
+ the tunnel always comes before the request reached the target, and "could not be reached" is true for each flag.
131
+
132
+ The Orchestrator route is different ([ADR-0002](0002-duplex-streaming-actor.md)). Its mapper adds the header
133
+ `x-jr2-custodian: unreachable` and the flags, and the Harness reads them to decide whether a pick may be sent again
134
+ (`menu.ts`). It says "never reached" only when each flag of the reply is `UF`, `UH`, `DF` or `URX`, and "not known"
135
+ when any other flag is there.
136
+
137
+ - Bodies stream both ways. A route has no total timeout; an idle stream has an hour; a client that goes away resets
138
+ the upstream request.
139
+
140
+ Every other target is tunneled, byte for byte, after the dial guard (below).
141
+
142
+ - **The engine choices.** The values and every credential decision are in a Lua filter. Its program is fixed
143
+ (`program.lua`); each pod's own `custodian.lua` is a generated `HELD` table — names, headers, paths and targets, never
144
+ a value — that ends with `dofile` of the program, so the program runs after the table it reads. At start the program
145
+ reads each value from `/etc/jr2/custodian/values/<NAME>`, takes off one trailing `\n` or `\r\n`, and stops the
146
+ start-up with a line that names the secret when a value is missing, empty, or has a byte outside `0x20`–`0x7E`.
147
+ Envoy's `credential_injector` is not used: it is alpha, it breaks on a secret file that ends in a newline (1.39.1),
148
+ and it replaces a whole header, where the swap must see the Stand-in. Values never enter a ConfigMap. Envoy runs one
149
+ worker, with hot restart off, and flushes its access log twice a second.
150
+ - **The log is a contract.** One line per decision on stdout, with the prefix `jr2.custodian` and `key=value` pairs:
151
+ `ready secrets=… hosts=N` (at the first probe), `action=intercept target=… secret=… method=… status=…`,
152
+ `action=tunnel target=…`, `action=refuse target=… reason=…`. Never a header value, a body, a path, a query, a value or
153
+ a Stand-in.
154
+ - **The dial guard, in two halves.** A tunnel never reaches a loopback, link-local (the cloud metadata address included,
155
+ and AWS's `fd00:ec2::254`) or unspecified address, or an IPv4-mapped form of one; it gets 403 and a `reason=guard`
156
+ line. The script refuses a target that SPELLS one, and `localhost` or a `.localhost` name, before anything is
157
+ resolved. The dynamic forward proxy filter saves the address it resolved a name to (`save_upstream_address`), and an
158
+ RBAC filter after it refuses that address when it is in `GUARDED_RANGES` — so a name that resolves to one is refused
159
+ like the literal. The address judged is the address dialed: both come from the one DNS cache entry.
160
+ - **Health.** `0.0.0.0:15021`, `GET /healthz`. The kubelet cannot reach a loopback listener, and a distroless image has
161
+ no shell for an exec probe. The Custodian's readiness probe is on it. Pod Ready needs every container ready, and the
162
+ operator holds a Sandbox's Ready on pod Ready, so no Turn is admitted before the Custodian serves. The Instance
163
+ Harness's rollout wait covers it the same way. It is the one listener a peer can reach, where the ADR-0058
164
+ NetworkPolicy is not enforced. So it is outside Envoy's global connection limit, holds 16 connections at most, and
165
+ closes a connection idle for 5 seconds: a peer's idle connections cannot stop the Menu or the egress.
166
+ - **The config surface** (`jr2.config.ts`,
167
+ [ADR-0018](0018-instance-agents-are-definitions-jr2-assembles-the-harness.md)): the top-level `secrets` list, whose
168
+ entry is
169
+ `{ name; value | valueFrom.secretKeyRef; hosts?; headers?, default `authorization`, `x-api-key`, `x-goog-api-key`, `api-key`; paths? }`,
170
+ and `harness.catalog` (a catalog provider id → `{ baseUrl }`). The list serves both sides
171
+ ([ADR-0068](0068-an-effector-holds-an-effects-secrets-and-signs-its-wire.md)). `harness.provider.apiKey` is an entry
172
+ too, implicitly: `JR2_PROVIDER_API_KEY`, bound to its `baseUrl`'s host. `loadConfig` checks the shape in both worlds.
173
+ - **Naming is placement: a pod gets the held secrets something in it names.** Placement is read, never declared
174
+ (`secretPlacementsOf`), and an entry's `hosts`, `headers` and `paths` are its binding — where the Custodian may put
175
+ the value — never its placement:
176
+
177
+ | Pod | The held secrets its Custodian holds |
178
+ | --------------------- | ------------------------------------------------------------------------------------------ |
179
+ | A Workspace's Sandbox | the names its `workspace()` lists under `secrets: [...]`, and the model keys of its Agents |
180
+ | The Instance Harness | the model keys of the `"none"` Agents it hosts, and nothing else |
181
+
182
+ A model key is claimed by the PROVIDER: an Agent names its model, and jr2 knows which keys that provider reads
183
+ (`MODEL_KEYS`; the custom provider's id for `JR2_PROVIDER_API_KEY`). An Agent's Turns run in the Workspace below which
184
+ it is declared, down to the next `workspace()`, unless it is `"none"`, whose Turns run at the Instance Harness
185
+ ([ADR-0031](0031-menu-only-agents-run-on-the-instance-harness.md)). A `"none"` Agent cannot execute code, so the
186
+ Instance Harness loses nothing by holding only model keys. Listing a model key under a `workspace()`'s `secrets` is
187
+ allowed, and `jr2 up` reports it as redundant. A name a `workspace()` lists that the list does not hold is refused,
188
+ with the line to add. An entry nothing names — no `workspace()`, no provider, no Effect — goes nowhere, and `jr2 up`
189
+ warns. `jr2 up` reports per pod:
190
+ `Workspace of workflow "coding": Custodian holds ANTHROPIC_API_KEY (provider), JIRA_TOKEN`.
191
+
192
+ `jr2 up` alone checks the claimed entries (`heldSecretsOf`), host-side, because `.env` values are absent in-cluster.
193
+ An entry no pod claims is not checked as a held secret:
194
+
195
+ | # | Refused |
196
+ | --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
197
+ | R1 | A known model key (`MODEL_KEYS`) in `harness.env`, or as a key of a Secret or ConfigMap `harness.envFrom` names. |
198
+ | R2 | A held-secret name that is `JR2_*` or a proxy or trust variable. |
199
+ | R3 | One name held twice, or held and set by `harness.env`. |
200
+ | R4 | Not exactly one of `value`/`valueFrom`; an empty value; a byte outside `0x20`–`0x7E`; leading or trailing space. |
201
+ | R5 | A claimed entry with no `hosts` where no default exists; a wildcard, userinfo, a non-`https` scheme, a bare-host path, a bad port; two secrets on one host in one header. |
202
+ | R6 | A `headers` entry that is not a lowercase token, or that frames or routes the request; a duplicate. |
203
+ | R7 | `provider.apiKey` with a non-`https` `baseUrl`; a non-`https` catalog `baseUrl`. |
204
+ | R8 | A catalog key that is not a known provider, or that is the custom provider's id. |
205
+ | R9 | Explicit `hosts` on a known model key that leave out where a carried Agent's provider sends its calls. |
206
+ | R10 | A `secretKeyRef` whose Secret or key does not exist. |
207
+ | R11 | While a secret is held: `harness.env` sets a proxy or trust variable. |
208
+ | R12 | A `paths` entry that does not start with `/`, or holds `..`, `;`, `\`, `%2e`, `%2f` or `%5c`. |
209
+ | R13 | A held value that also reaches the Harness: its Secret loaded by `harness.envFrom`, its key read by a `harness.env` `secretKeyRef`, or its literal set by `harness.env`. A kit Secret (`jr2-instance`, `jr2-held-*`, `jr2-instance-effector`, `jr2-effector-*`, `<sandbox>-effector`, `jr2-git-*`) named by either. |
210
+
211
+ Each message names the config path, never a value, and says what to write instead. A SigV4 or ADC credential in
212
+ `harness.env` is warned about, not refused. A known model key with no `hosts` is bound to the host its provider's
213
+ calls go to: the host of `harness.catalog.<id>.baseUrl` when set, else pi's catalog host. The key and the calls then
214
+ read one config and cannot drift. `MODEL_KEYS` is pi's data at the pin, written as plain data by `just model-keys` (a
215
+ script in `@jr2/harness`) at each pi bump, so the Orchestrator has no pi dependency. A test in `@jr2/harness` checks
216
+ the table against pi, and checks that the file is the script's exact text. So a pi bump that moves a provider, or a
217
+ hand edit of the file, fails the default gate until the recipe runs.
218
+
219
+ - **A catalog provider can move to a gateway.** pi has no env var for a catalog provider's base URL. It hands the
220
+ provider the model it is given. So `resolveModel` returns a copy of the catalog model with `baseUrl` replaced, for
221
+ every model of that provider. The catalog's other facts ride along. The model id sent upstream is the catalog id, so
222
+ the gateway must serve that name (a LiteLLM `model_name` alias).
223
+ - **Certificates.** `jr2 up` keeps a per-Instance CA in the `jr2-held-ca` Secret, which no pod mounts; it reads the CA
224
+ over the kube API. It issues one leaf per bound `host:port` into `jr2-held-tls` before a pod starts, so the CA's key
225
+ never enters a Harness pod. Keys are ECDSA P-256, written by a small DER writer in `@jr2/cli` over `node:crypto`: no
226
+ dependency, and no `openssl` on the host. `jr2 up` never rotates the CA; a human deletes the Secret to rotate it. A
227
+ leaf is reused while its SAN is the same, it chains to the current CA, and more than 30 days remain.
228
+ - **Trust.** The `jr2-ca` ConfigMap carries `ca.crt` (the user's `caBundle`), and while a secret is held: `extra.crt`
229
+ (`caBundle` + the jr2 CA, for `NODE_EXTRA_CA_CERTS`), `bundle.crt` (the roots + both, for the tools whose variable
230
+ replaces their store), and `upstream.crt` (the roots + `caBundle`, and never the jr2 CA, for the Custodian). "Roots"
231
+ are the Mozilla roots of the Node that runs `jr2 up`. The ConfigMap is applied server-side, because two copies of the
232
+ roots are more than client-side apply's annotation can hold.
233
+ - **Every Secret is applied server-side.** A client-side apply copies the whole object into its
234
+ `kubectl.kubernetes.io/last-applied-configuration` annotation, so the values of a Secret are then in plaintext for
235
+ each reader of its metadata. So `jr2 up` applies each Secret that it writes server-side: `jr2-instance`,
236
+ `jr2-harness-env`, `jr2-held-ca`, `jr2-held-tls`, `jr2-held-secrets-<pod>`, the Effector Secrets, the git Secrets, the
237
+ deploy key and the check pod's Secrets. The field manager is `jr2`, with `--force-conflicts`, because jr2 is the only
238
+ writer of these Secrets. A `stringData` goes as `data`, because a server-side apply owns the fields that it names, and
239
+ `stringData` is write-only. On a Secret that an older `jr2 up` applied client-side, the first server-side apply
240
+ migrates it: kubectl (1.26 and later) moves the fields of the client-side manager to `jr2`, so that apply removes the
241
+ annotation and each key that it does not name, and `jr2` is then the only manager (measured on kind with kubectl
242
+ 1.35.9). When that migration fails, kubectl only writes a warning on stderr. So after each apply of Secrets, a
243
+ fallback removes the annotation (`kubectl annotate … last-applied-configuration-`). The removal of an annotation that
244
+ is not there changes nothing, so the fallback needs no record of what was removed. When the fallback fails, `jr2 up`
245
+ writes one warning that names the Secrets, and the converge goes on. The rule is in the apply seam
246
+ (`kubectlAdmin.apply`), so a Secret that a later change adds is applied in the same way. The values go on stdin, never
247
+ in argv. No ConfigMap holds a value.
248
+ - **Objects, per pod.** The held manifest, the Custodian's files and the literal values are each a pod's own, from the
249
+ one derivation that places Effect secrets. A pod is keyed `instance` for the Instance Harness, and `ws-<digest>` for a
250
+ Workspace, the digest of its registered workflow and the placement route to its `workspace()` (the pod key below) —
251
+ the route the walk reads statically and the run host reads off the live actor at provision, so both name one pod
252
+ alike. Nothing rendered per pod lands in a shared object: `jr2 up` declares, and the pod's maker renders. `jr2-held`
253
+ (always) holds `held-<pod>.json` for each pod (what it holds: names, bindings, headers, never a value) and
254
+ `program.lua` once per Instance, so it grows by one manifest per Workspace. A pod's two rendered files, Envoy's
255
+ `envoy.json` and its `custodian.lua`, are in a ConfigMap of that pod's own: the Orchestrator renders a Sandbox's at
256
+ provision, from the manifest it reads, into `<sandbox>-custodian`, applied after the CR and owned by it like the
257
+ `<sandbox>-token` Secret, so it goes with the pod (the Orchestrator's Role may create and patch a ConfigMap, and no
258
+ more); `jr2 up` renders the Instance Harness's into `jr2-held-custodian`, applied with its StatefulSet and deleted
259
+ with it. A pod's `custodian-config` volume projects its own ConfigMap and `program.lua` from `jr2-held`. The reason:
260
+ every Kubernetes object stops at 1 MiB, and client-side apply's annotation at 256 KiB; one copy of Envoy's files per
261
+ pod in `jr2-held` passed that limit at nine Workspaces, and a dedupe or a server-side apply only moves the ceiling.
262
+ `jr2-held-secrets-<pod>` holds one pod's literal values, labeled `jr2.dev/held-pod`, and `jr2 up` deletes the one a
263
+ pod no longer needs. `jr2-held-tls` (every bound target's leaf) and `jr2-held-ca` are the Instance's. The Orchestrator
264
+ mounts `jr2-held` at `/etc/jr2/held` and reads the manifest of the Workspace it provisions, per provision — the
265
+ `jr2-images` pattern ([ADR-0038](0038-jr2-up-builds-every-image-it-deploys.md)); a Workspace `jr2 up` never walked has
266
+ none, and its provision is refused, naming `jr2 up`. A pod key is the workflow and the Workspace's placement route:
267
+ the `actors` slot keys down to its `workspace()`, a wrapper's body skipped, where a plain Machine invoked inline adds
268
+ xstate's generated invoke src for that hop (`xstate.invoke.<index>.<state id>`), held so it can never equal a slot
269
+ key. `jr2 up` computes it from the state node, and the provision reads it off the live actor tree, so both derive one
270
+ key. A key reached by slot keys alone is what it was. A wrapper's body adds no hop, so a `workspace()` reached through
271
+ another `workspace()`'s body (`workspace(workspace(...))`, `workspace(pool(workspace(...)))`) has the outer one's key:
272
+ `jr2 up` refuses that Machine, with or without a secret, since the two pods' secrets could not be kept apart, and
273
+ names the fix (the inner `workspace()` under a key of a plain Machine's `actors`, which has a route of its own); every
274
+ other Workspace has a key of its own. A Workspace reached through a Machine object handed to `spawnChild`, which no
275
+ walk can see, fails at provision with the slot form. One function, `custodianComposition`, builds what a pod gains
276
+ from its pod's manifest; `kubeSandbox` and `instanceHarnessObjects` both call it, so the two pod shapes cannot drift.
277
+ The Instance Harness's pod template carries `jr2.dev/held-digest`, an HMAC over its pod's held-secret inputs keyed
278
+ with the signing key, so a held-secret edit rolls it at the next `jr2 up`. Its Custodian reads each value and its
279
+ files once at start, so the inputs are its two rendered files, the program, a literal's value, a `valueFrom`'s
280
+ reference, and a digest of the value in the user's Secret, keyed with the signing key: `jr2 up` reads that Secret to
281
+ check the key exists, and keeps the digest in place of the value, so a value edited there rolls the pod too. A live
282
+ Sandbox keeps what its Custodian read at start.
283
+
284
+ ## Considered options
285
+
286
+ - **jr2's own proxy in Node, in a second container of the Adapter's image** (branch `held-secrets/node`). No new image
287
+ and no second language; jr2 then owns TLS-parsing code at the trust boundary, and needs its own tests for every edge
288
+ Envoy already handles. It keeps the Adapter.
289
+ - **Envoy in a container beside the Adapter** (branch `held-secrets/envoy-sidecar`). The same Envoy configuration, and
290
+ three containers per pod. Two containers hold two credentials, so a fault in one engine does not reach the other's
291
+ credential. This branch trades that split for one container fewer and one component fewer: the Custodian holds both.
292
+ - **The Orchestrator serves MCP, and Envoy forwards `/mcp/<iid>`.** Rejected. It reverses ADR-0013's "the Orchestrator
293
+ hosts no MCP", puts a transport and session handling back in the control plane, and keeps the MCP SDK on both ends —
294
+ to carry two plain HTTP calls. What ADR-0013 rejected about translation inside the Harness was the credential in that
295
+ container. The Custodian now holds it, so that reason is gone.
296
+ - **The Adapter relays model calls** (ADR-0058's first direction). Rejected: every provider needs its `baseUrl`
297
+ rewritten to the relay, and pi's catalog has no env var for a base URL, so the catalog could not follow.
298
+ - **Drop pi's catalog and name every model as a custom provider.** Rejected by the user: the catalog holds the model
299
+ facts the user would have to copy by hand.
300
+ - **iron-proxy.** It intercepts every host and cannot pass one through, and it makes leaves at runtime, so the CA's key
301
+ would be in the pod. The Sandbox Image is the user's, and its tools may not trust a jr2 CA.
302
+ - **mitmproxy, Squid.** mitmproxy streams SSE only if the addon asks, and its swap is code jr2 writes anyway; Squid
303
+ brings the GPL and a poor CVE record. Both would be a fourth image.
304
+ - **Envoy's `credential_injector`, or the value in a bootstrap Secret with `request_headers_to_add`.** Rejected: the
305
+ first is alpha and fails on a trailing newline; both replace a header without seeing the Stand-in, and the second
306
+ cannot hold a `secretKeyRef` value, which `jr2 up` never copies.
307
+ - **Intercept every host.** Rejected for the iron-proxy reason: a tool that does not trust the jr2 CA would then fail on
308
+ every host, not only on bound ones.
309
+
310
+ ## Consequences
311
+
312
+ - **What this closes, and what it does not.** The Agent cannot take a model key or the Sandbox token out of the pod. It
313
+ can still **spend** a key through the Custodian while its pod lives, on any path `paths` allows. A key with admin
314
+ rights can mint a new key, and a minted key is one the Agent can take away: use a key with no admin rights, and narrow
315
+ `paths`. The User Container shares the pod's loopback, so a human in it can spend a key the same way. That human
316
+ cannot read one either.
317
+ - **Response echo.** Responses pass unchanged. An upstream that echoes the full credential in a response hands it to the
318
+ Agent. Providers mask keys in errors; jr2 does not scrub responses.
319
+ - **Bypass.** A client that ignores `HTTPS_PROXY` goes direct and sends the Stand-in, which is worth nothing. It is not
320
+ blocked: egress control stays the cluster's decision (ADR-0058's open egress gap). The same holds for the control
321
+ plane: a caller that skips the Custodian and sends the Stand-in to the Orchestrator gets 401 on every route
322
+ (`auth.test.ts`).
323
+ - **Clients that do not trust the jr2 CA** (Java, a pinned client) fail TLS to a bound host. They fail closed and send
324
+ nothing. Unbound hosts still work for them.
325
+ - **`SSL_CERT_FILE` replaces an OpenSSL tool's store.** A private CA baked into the Sandbox Image, and not put in
326
+ `caBundle`, stops being trusted by curl, git and Python in a pod with a held secret. The fix is `caBundle`. The gain:
327
+ `caBundle` now reaches curl, git and Python, which [ADR-0020](0020-private-ca-trust-is-a-named-concept.md) recorded as
328
+ missing.
329
+ - **Signing credentials cannot be held.** A SigV4 or ADC credential signs the request with the secret, and a Stand-in
330
+ cannot be swapped into a signature. `jr2 up` warns. The fix is the Effector
331
+ ([ADR-0068](0068-an-effector-holds-an-effects-secrets-and-signs-its-wire.md)): an Effect secret is the real value, in
332
+ a container the Agent executes nothing in. Its `http` kind sends a bearer, a Basic pair or a header the author writes,
333
+ and the credentialed `command`
334
+ ([ADR-0067](0067-an-effect-is-an-actor-slot-and-runs-at-most-once-unless-marked-idempotent.md)) runs the signing tool
335
+ there (`terraform`, `aws`, `gcloud`), its child confined with Landlock from the Effector's key and every secret it
336
+ does not name. `ANTHROPIC_OAUTH_TOKEN` is held like a key, but pi sends a Stand-in as `x-api-key`, which an OAuth
337
+ token cannot ride; hold it as `ANTHROPIC_AUTH_TOKEN` (`Authorization: Bearer`) instead.
338
+ - **Rotation of a value** reaches a live Sandbox only when its pod is replaced.
339
+ - **The walk reads definitions, not Turns.** A pod's model keys come from the models its Agents' definitions name. A
340
+ Turn whose invocation overrides the model with another provider's (`AgentTurnInput.model`) finds no key for that
341
+ provider in its pod: the Harness container gets a Stand-in only for the names in its own pod's manifest, so that
342
+ provider's key variable is not set at all. The call then has no key (a catalog provider) or sends the custom
343
+ provider's placeholder `unused`, and it fails. This holds in a Workspace and in the Instance Harness alike. The fix is
344
+ a definition that names the provider; in a Workspace, the key under the `workspace()`'s `secrets` also works. The
345
+ Instance Harness has no such list: its pod holds only the model keys of the `"none"` Agents it hosts.
346
+ - **TLS-only bindings.** A key for a plain-HTTP endpoint is refused (R7). A LAN `llama-server --api-key` needs TLS and
347
+ its CA in `caBundle`, or no key.
348
+ - **No proxy chaining.** The Custodian cannot forward to a corporate proxy. R11 says so rather than fail silently.
349
+ - **Two Harness env shapes.** With no held secret in it, a pod gets the Custodian's address and the token's Stand-in and
350
+ no proxy, and its HTTPS goes direct. The Custodian runs in both shapes, because it carries the Menu.
351
+ - **The dial guard adds no confinement while egress is open.** The Agent can dial those addresses itself. The guard
352
+ matters when a cluster closes egress to everything but the Custodian: then the Custodian is not a way around it.
353
+ - **One container holds two kinds of credential.** A fault in Envoy's TLS or HTTP parsing is a fault in the container
354
+ that holds both the Sandbox token and the model keys. The branch with a separate Adapter keeps them apart.
355
+ - **The Harness container now speaks the Orchestrator's two agent routes.** The Agent could always make those calls (the
356
+ Adapter would forward any MCP call it asked for); now the Harness makes them itself. Nothing is judged in the pod.
357
+ - **`jr2-upload-pack` carries the Stand-in as a constant.** The User Container gets no env (ADR-0005), and asks like any
358
+ other seat. A test pins the Go literal to `standIn("JR2_SANDBOX_TOKEN")`.
359
+ - **The Custodian's image is a Pinned image.** It is deployed and never built, which
360
+ [ADR-0038](0038-jr2-up-builds-every-image-it-deploys.md) now states. `kitRegistry` re-homes it and `jr2 kit push`
361
+ mirrors it ([ADR-0044](0044-kit-images-live-at-a-canonical-home-a-self-host-mirrors-it.md)). The image sweep never
362
+ takes it, because jr2 never stamped it ([ADR-0039](0039-image-garbage-collects-by-reachability.md)). A kind node pulls
363
+ it from its registry, so the `@kind` tier needs network to `docker.io`. `just kit-push` does not seed it.
364
+ - **Where the Custodian suite runs.** `just custodian-test`: node's test runner against the pinned image under docker,
365
+ on the host network with free ports, with local HTTPS upstreams under their own CA. It is opt-in, like `@kind`,
366
+ because it needs docker ([ADR-0010](0010-bdd-acceptance-tests.md)), and the release job runs it on every tag
367
+ ([ADR-0055](0055-a-release-is-a-pushed-tag-and-the-publish-guard-is-the-credential.md)): the image moves only with its
368
+ pin, but the configuration jr2 renders moves with any release. About 35 seconds. It checks the swap, the strip, 403
369
+ with no Stand-in, 405 on a method, 400 on an underscore header, a header `Connection` names, two `Host` headers, idle
370
+ connections on the health listener, 421, 400 and 403 on paths, the tunnel, the dial guard by every spelling and by
371
+ resolved name, 502 on an untrusted upstream, SSE timing, abort, a 302, the start-up refusal, the log, and the
372
+ Anthropic SDK through `HTTPS_PROXY`. The first SSE event arrives through the Custodian about half a millisecond later
373
+ than direct.
374
+ - **The `@kind` tier's model wants a key.** Its fake provider serves HTTPS under the tier's CA and refuses a request
375
+ without the key, which is `harness.provider.apiKey` — so every turn in the tier crosses a Custodian, in both
376
+ placements. A Rule proves the key is in no env and no file of either Harness container.
377
+ - **The provider preflight crosses a Custodian.** A provider's key is held. So `jr2 up` checks the provider from a pod
378
+ with a Custodian that holds its keys, and the stock Harness's `provider-preflight.ts`
379
+ ([ADR-0019](0019-one-converging-command-against-the-current-context.md)). The check pod reads its own copies of the
380
+ held objects. The check runs before `jr2 up` writes an object that the Instance or the operator reads. A literal key
381
+ is in the check pod's own Secret while the pod lives. A wrong or missing key, no egress to the bound host, and a
382
+ gateway that does not answer fail the converge, not a Turn.
383
+ - **The CA has no name constraints.** Constraints would force a new CA whenever the bound hosts change, and that breaks
384
+ a live pod whose Custodian restarts.
@@ -0,0 +1,111 @@
1
+ # A Workspace states its Size, and jr2 reserves all of it
2
+
3
+ No jr2 pod asked Kubernetes for anything. `Sandbox.spec.resources` existed and nothing filled it; the Custodian, the
4
+ Instance Harness and the Orchestrator set nothing either. Every pod was BestEffort, so the scheduler packed Sandboxes
5
+ without limit, the cluster autoscaler saw no demand, and a full cluster showed up as OOMKill and eviction instead of
6
+ Pending — an eviction takes `/work` with it (ADR-0021). The scaling review (2026-09-27) and a lab of real workloads on
7
+ the home cluster (2026-09-28: Playwright, `tsc` + vitest on hono, `cargo build` of ripgrep; one run each) measured what
8
+ a Sandbox uses. The owner plans to move Workspaces onto Agent Substrate later, so the shape is chosen to move there
9
+ unchanged.
10
+
11
+ ## Decision
12
+
13
+ - **A Workspace states one Size: a ceiling for its whole Sandbox, cpu and memory, nothing else.** The shape is Agent
14
+ Substrate's `ActorTemplate.resources`: a Kubernetes `ResourceRequirements` of which only `limits.cpu` and
15
+ `limits.memory` are accepted; `requests` and any other key are refused where they are read.
16
+ `workspace(body, { resources: { limits: { memory: "3Gi", cpu: "2" } } })`.
17
+ - **The Size is a fact of the Machine**, like its Sandbox Image (ADR-0037, ADR-0049): the author knows the toolchain,
18
+ the runner, the worker count. It is never per run. The resolution chain, most specific first:
19
+ `customize(machine, { resources })` (the composer, whose Repo may be bigger than the author's) → the `workspace()` →
20
+ `sandbox.resources` in `jr2.config.ts` (the Instance's default, applied **only** to a Workspace that states none —
21
+ never an override of a stated Size, so each fact has one override path) → the kit default.
22
+ - **jr2 reserves the whole Size.** Memory request = limit and cpu request = limit: the Sandbox pod is Guaranteed. Memory
23
+ pressure never picks a Workspace, and placement accounts the way Substrate's does. Density is paid for here and
24
+ recovered later by suspend (the scaling review's R15), not by overcommit.
25
+ - **The CPU limit is enforced.** Nothing reads a CPU request (the cgroup shows `cpu.max = max`), while Node's
26
+ `availableParallelism`, Go 1.25, Rust and cargo, the JVM, .NET and every Substrate runtime read the limit. Throttling
27
+ is the accepted cost.
28
+ - **Tools that read neither the limit nor affinity get hints**, from the Downward API (`limits.cpu`, divisor 1):
29
+ `JR2_CPUS`, `OMP_NUM_THREADS` (which also makes `nproc` answer N), `PYTHON_CPU_COUNT`, `GOMAXPROCS`. `os.cpus()` still
30
+ reports the node, and Playwright sizes its workers from it alone, so a Machine passes `--workers=$JR2_CPUS`; the Agent
31
+ is told its CPU count by the Briefing ([ADR-0062](0062-jr2-briefs-the-agent-on-its-seat-never-on-its-task.md)), never
32
+ by the Frame.
33
+ - **jr2 splits the Size inside the pod.** The Custodian takes a fixed kit slice (64Mi, 50m). The User Container takes
34
+ its split if the Machine states one — `user` widens from a string to
35
+ `{ image, resources?: { limits: { memory?, cpu? } } }` (ADR-0005 foresaw the widening) — and otherwise shares the pod
36
+ budget with no limit of its own. The Sandbox Effector, in a pod that has one
37
+ ([ADR-0068](0068-an-effector-holds-an-effects-secrets-and-signs-its-wire.md)), has no split and no limit of its own
38
+ and shares the pod budget the same way. The Harness container gets the rest as its own limit, so an OOM stays per
39
+ container where it can. Pod-level resources (KEP-2837) hold the whole-pod ceiling. A server with the PodLevelResources
40
+ gate off drops them without a word, so `jr2 up` dry-runs a pod that states them and refuses the cluster when they are
41
+ gone. `jr2 up` refuses a split that leaves the Harness below its floor (256Mi) and names the line.
42
+ - **Kit numbers.** Default Size: memory 2Gi, cpu 1. Node gives each process a V8 heap of about 55% of its container
43
+ limit, and hono's `tsc` aborted with a heap error below about 1.1Gi, so 1Gi fails a common task. 2Gi covers `tsc` +
44
+ vitest, `cargo build` and Playwright at 2 workers; Playwright at 4 workers and large monorepos state a Size. No
45
+ `NODE_OPTIONS` heap raise: a V8 abort is a tool error the Agent reads, while a bigger heap turns it into a kernel
46
+ kill.
47
+ - **`/dev/shm` is a memory-backed emptyDir in the Harness container only**, `sizeLimit` = 25% of the Harness share
48
+ (minimum 64Mi). It reserves nothing extra (shm is charged inside the Harness limit), and a full shm is ENOSPC or
49
+ SIGBUS — a tool error, never an OOM (lab). The Harness clears it at start: shm files outlive a container restart and
50
+ stay charged to the pod, which crash-looped the restarted container in the lab. There is no `shm` field; a Machine
51
+ that needs more raises its Size.
52
+ - **`/work` disk is a kit-fixed ephemeral-storage request of 10Gi on the Harness container, with no cap.** The request
53
+ places the pod and ranks it last under node disk pressure; no `sizeLimit` and no limit, because passing one evicts the
54
+ pod and loses `/work` (T12). Disk is not part of the Size: Substrate's template size is cpu and memory only.
55
+ - **Priority.** `jr2 up` creates two cluster-scoped PriorityClasses beside the CRDs: `jr2-control` (100000,
56
+ PreemptLowerPriority) for the Orchestrator, the operator and the Repo cache agent — small, fixed and needed by every
57
+ Sandbox — and the Instance Effector ([ADR-0068](0068-an-effector-holds-an-effects-secrets-and-signs-its-wire.md)),
58
+ small and fixed; `jr2-sandbox` (1000, preemptionPolicy `Never`) for Sandboxes and the Instance Harness — a pod of
59
+ ordinary priority cannot preempt a live Workspace, and a waiting Sandbox evicts nobody. Both stay far below the system
60
+ classes. A cluster with its own scheme names its classes in `priorityClasses: { control?, sandbox? }`.
61
+ - **Voluntary disruption.** A Sandbox pod and the Instance Harness pod carry
62
+ `cluster-autoscaler.kubernetes.io/safe-to-evict: "false"` and `karpenter.sh/do-not-disrupt: "true"`: moving either
63
+ loses work (`/work`; live conversations). The Orchestrator, the operator, the Repo cache agent and the Instance
64
+ Effector carry neither — a restarted Orchestrator re-attaches (ADR-0007), and a stopping Effector finishes its Effects
65
+ first. No PodDisruptionBudget: `maxUnavailable: 0` blocks node upgrades without end. A drain ends in `workspace.lost`,
66
+ and the body's policy decides (ADR-0021).
67
+ - **jr2's own pods are kit-sized, with no config key**: memory request = limit, a CPU request, no CPU limit (a
68
+ provisioning burst needs CPU, and no Substrate move applies to them). Orchestrator 500m / 1Gi. Instance Harness 250m /
69
+ 1Gi (it frees every conversation no live run holds — ADR-0031). Operator 50m / 256Mi (it caches every Pod in the
70
+ cluster). Instance Effector 50m / 128Mi. Repo cache agent 50m, memory request 128Mi and limit 1Gi — the one exception,
71
+ since `git index-pack` spikes and its OOM costs only a refetch.
72
+
73
+ ## Considered options
74
+
75
+ - **Sizes as named classes in `jr2.config.ts`, which the Machine names** (`workspace(body, { class: "browser" })`, the
76
+ config maps the class to resources). Rejected: it breaks packaging. Two packages each name `browser` and mean
77
+ different sizes, and the consumer must supply numbers only the package's author knew. ADR-0052's named class stays
78
+ placement only — node labels are the cluster's facts; a memory need is the Machine's.
79
+ - **Raw `ResourceRequirements`, all of it.** Rejected: the author could set a request below the limit, leave out the
80
+ memory limit, or reach fields Substrate refuses, and a Machine would depend on something the move drops.
81
+ - **A jr2 shape** (`{ memory, memoryLimit?, cpu }`). Rejected for the same reason in the other direction: one more
82
+ translation at the move, for no gain the Substrate subset does not give.
83
+ - **Burstable: reserve the working peak, allow bursts to the limit.** About twice the density. Rejected: with no
84
+ suspend, several Workspaces bursting at once is node pressure, and pressure evicts a whole pod with its `/work`.
85
+ - **Reserve CPU, never throttle.** Chosen first, then reversed: no mechanism tells a tool its CPU request (the lab tried
86
+ affinity, CPU Manager, LXCFS, gVisor, Kata and `LD_PRELOAD` shims; all but affinity need a limit, and affinity misses
87
+ `os.cpus()` and lets pods pin the same cores), while every container-aware runtime reads the limit.
88
+ - **An `shm` field, or a `/work` disk field, on the Size.** Deferred: Substrate has neither. A jr2 field beside the Size
89
+ can come when a real Machine needs one.
90
+ - **`/dev/shm` without a `sizeLimit`, or the container's 64Mi default.** Rejected: unbounded, a full shm OOM-kills the
91
+ container and poisons its restart; 64Mi crashed Chromium renderers when Playwright's `--disable-dev-shm-usage` was
92
+ off.
93
+
94
+ ## Consequences
95
+
96
+ - ADR-0005 is rewritten in place: containers now carry limits from the Size, the User Container may take a split, and
97
+ `/dev/shm` joins the Harness container's mounts.
98
+ - ADR-0052 gains `sandbox.resources` beside `nodeSelector` and `tolerations`; its reserved named class stays placement
99
+ only, so its `node:` name stays.
100
+ - An OOM in the Harness container still kills the Harness with the Agent's processes (cgroup v2 group kill). Containing
101
+ it is ADR-0061.
102
+ - A Workspace parked on a Gate holds its whole Size. That is the density problem R15 exists for.
103
+ - On a move to Substrate, a Workspace with a User Container split needs a microVM pool: Substrate accepts per-container
104
+ limits on microVM runtimes only, and gVisor refuses them.
105
+ - `jr2 up` warns — never refuses — when no current Sandbox node could hold a Workspace's Size (the ADR-0052 stance: the
106
+ node set moves).
107
+ - GLOSSARY.md gains **Size**.
108
+ - The kit-sized control pods ask a node for about 900m a scenario before any Sandbox, so the `@kind` tier on a 4-vCPU
109
+ hosted runner runs two workers, not the profile's four (`just e2e-kind 2` in the release job). v0.4.0's release run
110
+ measured the alternative: four workers left every Sandbox Pending on `Insufficient cpu`. The requests are the kit's
111
+ and are not lowered for a runner.
@@ -0,0 +1,83 @@
1
+ # A memory kill ends the Agent's processes, not the Harness
2
+
3
+ The Agent runs its programs (Playwright and its Chromium, `tsc`, `cargo`) through the `bash` Working tool, so they are
4
+ children of the Harness, in the Harness container's one cgroup (ADR-0005, ADR-0037). On cgroup v2 the kubelet sets
5
+ `memory.oom.group`, so when those programs pass the container's limit (ADR-0060) the kernel kills the whole container:
6
+ the Harness dies with them, its conversation with it, and the Turn faults as "conversation lost" (lab 2026-09-28; chaos
7
+ test T11). Worse, the bash tool that pi-agent-core 0.82.1 shipped reported a command killed by a signal as success
8
+ (`exitCode: code ?? 0`), so when a program was killed the Agent saw its output stop partway and read it as a pass.
9
+
10
+ **The Agent's processes** are every process in the Harness container except the Harness and its init: the tree of the
11
+ running `bash` call, anything the Agent left running from earlier calls (a dev server, a watcher), and a human's
12
+ `kubectl exec` session.
13
+
14
+ ## Decision
15
+
16
+ Layers, each a backstop for the one before:
17
+
18
+ 1. **A process killed by a signal is an error.** jr2's bash tool reads the shell's exit: a signal death, of the command
19
+ or of the shell, reports the signal and never exit 0. A Harness conformance test holds it (ADR-0027).
20
+ 2. **`tini` is PID 1 in the Harness container**, shipped on `/opt/jr2`, so killed trees leave no zombies. Not
21
+ `shareProcessNamespace` — ADR-0005 keeps it off for good.
22
+ 3. **The `bash` Working tool starts each command with `oom_score_adj=1000`**, inherited by its children. It needs no
23
+ privilege, and on a node whose owner set the kubelet's `singleProcessOOMKill: true` the kernel then picks the Agent's
24
+ process, not the Harness.
25
+ 4. **The Harness guards its own cgroup.** Every 10–20 ms it reads anon + shmem from `memory.stat` against `memory.max`
26
+ (never `memory.current`, which counts reclaimable page cache). Near the limit it kills all of the Agent's processes,
27
+ checks usage again, clears `/dev/shm`, and the running `bash` call returns
28
+ `killed: memory limit (peak X of Y); use fewer workers or a larger Size`. The Agent hears it in the same Turn and
29
+ conversation, and can retry smaller. The guard polls from the Harness's start, not only during a call. A process that
30
+ an earlier Turn left running (a dev server, a watcher) can pass the limit while no Submission runs. For example, the
31
+ workflow waits at a human's Gate. Then no call returns the text. The Harness keeps every kill for the Orchestrator to
32
+ read, and a read removes nothing. The next Agent that works in that Workspace is told (ADR-0062).
33
+ 5. **The kernel's group kill is the last backstop.** The Orchestrator reads `lastState.terminated.reason: OOMKilled` and
34
+ the fault reason starts with the fixed prefix `memory limit`, not `Turn lost` (any other restart, ADR-0021) or
35
+ "conversation lost" (a 404). The fault reaches the Machine (ADR-0016); the next Agent in that Workspace is told
36
+ through the Briefing ([ADR-0062](0062-jr2-briefs-the-agent-on-its-seat-never-on-its-task.md)), never through the
37
+ Frame (ADR-0057). The kit's Machines pass the reason into their retry Turn as the example for authors. A kernel kill
38
+ while no Submission runs faults no Turn, because the restarted Harness rebuilds its conversations. The operator
39
+ publishes the container's last end on the Sandbox all the same, so before each admission into the Workspace the
40
+ Orchestrator reads it from the Sandbox watch, and a last end `OOMKilled` that no notice told yet tells the next Agent
41
+ (ADR-0062).
42
+
43
+ The guard is best-effort and says so: a fast enough spike beats any poll. The lab won 35/35 on arm64 at 50 ms, 6/6 on
44
+ amd64 at 10 ms (0.8% of a core), and only half on amd64 at 50 ms; it fired at 90–96% of the limit. Layer 5 covers what
45
+ it misses.
46
+
47
+ **The planned successor is a child cgroup for the Agent's processes** (`memory.max` below the container's, its own
48
+ `oom.group`): exact, no race, the Harness always survives (lab: 3/3 on kind). It needs a writable cgroupfs, which today
49
+ only a node owner grants (containerd ≥ 2.1 `cgroup_writable` behind a RuntimeClass, not offered on managed clusters).
50
+ KEP-5474 adds `securityContext.cgroupOptions.mountMode: Writable` so the pod asks for it itself; it is alpha, targeted
51
+ at v1.38. When it is enabled by default, jr2 sets the field on the Harness container and the child cgroup replaces layer
52
+ 4's polling.
53
+
54
+ ## Considered options
55
+
56
+ - **Accept the group kill.** Rejected as the whole answer: every memory kill becomes a Turn fault, and the Agent never
57
+ learns why, so a retry repeats it.
58
+ - **The Harness in its own container, with an exec server in the Sandbox Image's container running the Agent's
59
+ processes.** A real cgroup edge, but it reverses "Working tools execute in the Harness container" (ADR-0037) for a
60
+ problem layers 4 and the successor solve inside it.
61
+ - **Adopt the child cgroup now, used when detected.** Rejected: a path almost no cluster runs is a path nobody tests,
62
+ and the guard must be solid on its own anyway.
63
+ - **MemoryQoS `memory.high`.** Rejected: the lab saw a stall of over 9 minutes with no kill and `kubectl exec` hung.
64
+ - **earlyoom, systemd-oomd, oomd.** Rejected: they read the host's `/proc/meminfo`, need systemd, or need PSI triggers,
65
+ which a read-only cgroupfs refuses. The guard borrows their idea.
66
+ - **Kill only the running call's process group.** Rejected: `setsid` children escape it (lab), and a dev server left
67
+ from an earlier call may hold the memory.
68
+ - **Raise Node's heap for the Agent's processes** (`NODE_OPTIONS`). Rejected in ADR-0060: it turns a readable V8 abort
69
+ into a kernel kill.
70
+
71
+ ## Consequences
72
+
73
+ - The guard's threshold comes from the Harness container's limit, so it needs ADR-0060's limits first; layer 1 does not
74
+ and is fixed at once.
75
+ - The kill set includes a human's exec session open at that moment; the group kill would end it too. On the Instance
76
+ Harness that session is the only process the guard can kill, because a Menu-only Agent has no Working tools. No notice
77
+ follows: that pod holds no Workspace (ADR-0031, ADR-0062).
78
+ - A kernel kill (layer 5) while no Submission runs is told to the next Turn through the container's last end, which the
79
+ watch keeps until the container ends again. So two kernel kills before the next admission are told as one, and a
80
+ kernel kill that a later end of another kind replaces is not told. The guard fires at 90% of the limit, before the
81
+ kernel, so this window is the guard's misses alone.
82
+ - A memory fault is named in `jr2 status` and on the feed, not `Turn lost` (a restart for any other reason) or
83
+ "conversation lost" (a 404).