@jr2/orchestrator 0.3.0 → 0.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +20 -2
- package/bin/server.ts +7 -4
- package/console/canvas.ts +27 -14
- package/console/components/app.ts +12 -1
- package/console/components/fleet.ts +52 -6
- package/console/components/machine-pane.ts +128 -2
- package/console/components/nav.ts +1 -1
- package/console/components/schema-form.ts +12 -4
- package/console/json-field.ts +23 -0
- package/console/main.ts +122 -39
- package/console/store.ts +321 -21
- package/console/style.css +111 -0
- package/docs/agents-block.md +5 -0
- package/docs/concepts/agents-and-turns.md +442 -0
- package/docs/concepts/architecture.md +230 -0
- package/docs/concepts/charts.md +143 -0
- package/docs/concepts/composition.md +357 -0
- package/docs/concepts/converge.md +254 -0
- package/docs/concepts/durable-runs.md +189 -0
- package/docs/concepts/effects.md +603 -0
- package/docs/concepts/gates.md +208 -0
- package/docs/concepts/machines-and-workflows.md +289 -0
- package/docs/concepts/pools-and-sources.md +230 -0
- package/docs/concepts/secrets-and-trust.md +222 -0
- package/docs/concepts/the-menu.md +378 -0
- package/docs/concepts/workspaces.md +678 -0
- package/docs/get-started/a-model-decides.md +563 -0
- package/docs/get-started/an-agent-in-a-sandbox.md +381 -0
- package/docs/get-started/connect-a-model.md +248 -0
- package/docs/get-started/create-an-instance.md +194 -0
- package/docs/get-started/have-your-assistant-write-a-machine.md +922 -0
- package/docs/get-started/install.md +126 -0
- package/docs/guides/add-tools-to-the-sandbox.md +295 -0
- package/docs/guides/answer-a-gate.md +304 -0
- package/docs/guides/bind-open-parts.md +223 -0
- package/docs/guides/cancel-a-run.md +146 -0
- package/docs/guides/clean-up.md +206 -0
- package/docs/guides/fix-a-failed-converge.md +361 -0
- package/docs/guides/give-the-cluster-git-credentials.md +222 -0
- package/docs/guides/iterate-on-a-machine-on-kind.md +190 -0
- package/docs/guides/let-an-agent-write-a-chart.md +226 -0
- package/docs/guides/look-inside-a-sandbox.md +174 -0
- package/docs/guides/mirror-the-kit-images.md +176 -0
- package/docs/guides/move-from-kind-to-a-shared-cluster.md +302 -0
- package/docs/guides/open-the-console.md +91 -0
- package/docs/guides/publish-a-machine-as-an-npm-package.md +350 -0
- package/docs/guides/push-and-sign-a-branch.md +486 -0
- package/docs/guides/run-a-chart-by-name.md +270 -0
- package/docs/guides/run-a-command-or-call-a-url.md +544 -0
- package/docs/guides/run-one-worker-per-item.md +321 -0
- package/docs/guides/set-an-agents-access-and-dials.md +375 -0
- package/docs/guides/size-a-workspace.md +204 -0
- package/docs/guides/start-a-run-from-a-run.md +237 -0
- package/docs/guides/start-a-run.md +207 -0
- package/docs/guides/upgrade-the-kit-version.md +240 -0
- package/docs/guides/use-a-hosted-provider.md +208 -0
- package/docs/guides/use-a-local-model.md +313 -0
- package/docs/guides/watch-a-run.md +326 -0
- package/docs/guides/write-a-workflow.md +318 -0
- package/docs/index.md +44 -0
- package/docs/llms.txt +216 -0
- package/docs/reference/authoring-surface/agent.md +128 -0
- package/docs/reference/authoring-surface/answer.md +140 -0
- package/docs/reference/authoring-surface/arg.md +60 -0
- package/docs/reference/authoring-surface/basic.md +62 -0
- package/docs/reference/authoring-surface/bearer.md +53 -0
- package/docs/reference/authoring-surface/chartschema.md +23 -0
- package/docs/reference/authoring-surface/chartschemafor.md +48 -0
- package/docs/reference/authoring-surface/command.md +191 -0
- package/docs/reference/authoring-surface/customize.md +64 -0
- package/docs/reference/authoring-surface/defineconfig.md +41 -0
- package/docs/reference/authoring-surface/defineevent.md +114 -0
- package/docs/reference/authoring-surface/dispatch.md +126 -0
- package/docs/reference/authoring-surface/fail.md +83 -0
- package/docs/reference/authoring-surface/file.md +122 -0
- package/docs/reference/authoring-surface/frompick.md +277 -0
- package/docs/reference/authoring-surface/http.md +159 -0
- package/docs/reference/authoring-surface/isfailure.md +58 -0
- package/docs/reference/authoring-surface/jr2setup.md +237 -0
- package/docs/reference/authoring-surface/open.md +74 -0
- package/docs/reference/authoring-surface/openbody.md +86 -0
- package/docs/reference/authoring-surface/paramsof.md +125 -0
- package/docs/reference/authoring-surface/pool.md +186 -0
- package/docs/reference/authoring-surface/push.md +112 -0
- package/docs/reference/authoring-surface/secret.md +118 -0
- package/docs/reference/authoring-surface/sentby.md +83 -0
- package/docs/reference/authoring-surface/source.md +51 -0
- package/docs/reference/authoring-surface/text.md +62 -0
- package/docs/reference/authoring-surface/type-agentdeclaration.md +27 -0
- package/docs/reference/authoring-surface/type-agentdefinition.md +80 -0
- package/docs/reference/authoring-surface/type-agentlogic.md +15 -0
- package/docs/reference/authoring-surface/type-agentslots.md +12 -0
- package/docs/reference/authoring-surface/type-answeraction.md +12 -0
- package/docs/reference/authoring-surface/type-arg.md +19 -0
- package/docs/reference/authoring-surface/type-argnames.md +12 -0
- package/docs/reference/authoring-surface/type-binding.md +62 -0
- package/docs/reference/authoring-surface/type-bindingof.md +12 -0
- package/docs/reference/authoring-surface/type-bodyacceptsdoor.md +15 -0
- package/docs/reference/authoring-surface/type-bodyacceptsslots.md +17 -0
- package/docs/reference/authoring-surface/type-ceilings.md +41 -0
- package/docs/reference/authoring-surface/type-chainlink.md +19 -0
- package/docs/reference/authoring-surface/type-chartaction.md +46 -0
- package/docs/reference/authoring-surface/type-chartagent.md +28 -0
- package/docs/reference/authoring-surface/type-chartevent.md +20 -0
- package/docs/reference/authoring-surface/type-chartguard.md +20 -0
- package/docs/reference/authoring-surface/type-chartinvoke.md +28 -0
- package/docs/reference/authoring-surface/type-chartjson.md +14 -0
- package/docs/reference/authoring-surface/type-chartstate.md +46 -0
- package/docs/reference/authoring-surface/type-charttext.md +40 -0
- package/docs/reference/authoring-surface/type-charttransition.md +22 -0
- package/docs/reference/authoring-surface/type-charttransitions.md +14 -0
- package/docs/reference/authoring-surface/type-chartvalue.md +15 -0
- package/docs/reference/authoring-surface/type-childat.md +12 -0
- package/docs/reference/authoring-surface/type-childslots.md +12 -0
- package/docs/reference/authoring-surface/type-commandenv.md +15 -0
- package/docs/reference/authoring-surface/type-commandinput.md +19 -0
- package/docs/reference/authoring-surface/type-commandoptions.md +78 -0
- package/docs/reference/authoring-surface/type-commandoutput.md +104 -0
- package/docs/reference/authoring-surface/type-customize.md +109 -0
- package/docs/reference/authoring-surface/type-dispatchdefinition.md +39 -0
- package/docs/reference/authoring-surface/type-dispatchinput.md +14 -0
- package/docs/reference/authoring-surface/type-dispatchopenparts.md +14 -0
- package/docs/reference/authoring-surface/type-dispatchoutput.md +65 -0
- package/docs/reference/authoring-surface/type-duration.md +15 -0
- package/docs/reference/authoring-surface/type-effectbinding.md +93 -0
- package/docs/reference/authoring-surface/type-effectlogic.md +12 -0
- package/docs/reference/authoring-surface/type-envopenparts.md +14 -0
- package/docs/reference/authoring-surface/type-eventaudience.md +27 -0
- package/docs/reference/authoring-surface/type-eventdef.md +45 -0
- package/docs/reference/authoring-surface/type-eventfrom.md +16 -0
- package/docs/reference/authoring-surface/type-eventsemantics.md +26 -0
- package/docs/reference/authoring-surface/type-execoutcome.md +34 -0
- package/docs/reference/authoring-surface/type-failslot.md +14 -0
- package/docs/reference/authoring-surface/type-failure.md +19 -0
- package/docs/reference/authoring-surface/type-failureinput.md +20 -0
- package/docs/reference/authoring-surface/type-filebody.md +41 -0
- package/docs/reference/authoring-surface/type-frominput.md +14 -0
- package/docs/reference/authoring-surface/type-hostinjectedinput.md +58 -0
- package/docs/reference/authoring-surface/type-httpdefinition.md +101 -0
- package/docs/reference/authoring-surface/type-httpinput.md +12 -0
- package/docs/reference/authoring-surface/type-httpmethod.md +14 -0
- package/docs/reference/authoring-surface/type-httpopenparts.md +12 -0
- package/docs/reference/authoring-surface/type-httpoutput.md +89 -0
- package/docs/reference/authoring-surface/type-httprequestparts.md +33 -0
- package/docs/reference/authoring-surface/type-jr2config.md +126 -0
- package/docs/reference/authoring-surface/type-lends.md +38 -0
- package/docs/reference/authoring-surface/type-logicat.md +12 -0
- package/docs/reference/authoring-surface/type-markedif.md +12 -0
- package/docs/reference/authoring-surface/type-mergedactors.md +12 -0
- package/docs/reference/authoring-surface/type-namedsecret.md +25 -0
- package/docs/reference/authoring-surface/type-noslotspec.md +26 -0
- package/docs/reference/authoring-surface/type-noslotworkspaceoptions.md +32 -0
- package/docs/reference/authoring-surface/type-openbodydeclaration.md +40 -0
- package/docs/reference/authoring-surface/type-openbodylogic.md +14 -0
- package/docs/reference/authoring-surface/type-openbodyoutput.md +42 -0
- package/docs/reference/authoring-surface/type-openbodyslot.md +18 -0
- package/docs/reference/authoring-surface/type-opencommandinput.md +17 -0
- package/docs/reference/authoring-surface/type-openeffect.md +12 -0
- package/docs/reference/authoring-surface/type-openeffectslots.md +12 -0
- package/docs/reference/authoring-surface/type-openof.md +12 -0
- package/docs/reference/authoring-surface/type-opensecret.md +23 -0
- package/docs/reference/authoring-surface/type-opensecretmark.md +12 -0
- package/docs/reference/authoring-surface/type-permissivenoslotworkspaceoptions.md +31 -0
- package/docs/reference/authoring-surface/type-permissiveworkspaceoptions.md +32 -0
- package/docs/reference/authoring-surface/type-poolmachine.md +28 -0
- package/docs/reference/authoring-surface/type-poolspec.md +60 -0
- package/docs/reference/authoring-surface/type-pushinput.md +15 -0
- package/docs/reference/authoring-surface/type-pushoptions.md +79 -0
- package/docs/reference/authoring-surface/type-pushoutput.md +124 -0
- package/docs/reference/authoring-surface/type-pushwireoutput.md +12 -0
- package/docs/reference/authoring-surface/type-quantity.md +14 -0
- package/docs/reference/authoring-surface/type-reached.md +12 -0
- package/docs/reference/authoring-surface/type-reposlot.md +70 -0
- package/docs/reference/authoring-surface/type-reposlotsof.md +16 -0
- package/docs/reference/authoring-surface/type-sandboxoptions.md +77 -0
- package/docs/reference/authoring-surface/type-secret.md +25 -0
- package/docs/reference/authoring-surface/type-sender.md +19 -0
- package/docs/reference/authoring-surface/type-sentbyguards.md +12 -0
- package/docs/reference/authoring-surface/type-size.md +85 -0
- package/docs/reference/authoring-surface/type-slothandles.md +32 -0
- package/docs/reference/authoring-surface/type-sourcenextresult.md +17 -0
- package/docs/reference/authoring-surface/type-sourcespec.md +44 -0
- package/docs/reference/authoring-surface/type-syncstate.md +12 -0
- package/docs/reference/authoring-surface/type-text.md +19 -0
- package/docs/reference/authoring-surface/type-textpart.md +24 -0
- package/docs/reference/authoring-surface/type-thinkinglevel.md +39 -0
- package/docs/reference/authoring-surface/type-toparameterizedobject.md +14 -0
- package/docs/reference/authoring-surface/type-toprovidedactor.md +14 -0
- package/docs/reference/authoring-surface/type-unknown.md +21 -0
- package/docs/reference/authoring-surface/type-usercontainer.md +77 -0
- package/docs/reference/authoring-surface/type-withoutemptyindex.md +20 -0
- package/docs/reference/authoring-surface/type-workflowevent.md +46 -0
- package/docs/reference/authoring-surface/type-workspaceaccess.md +16 -0
- package/docs/reference/authoring-surface/type-workspaced.md +94 -0
- package/docs/reference/authoring-surface/type-workspacehandles.md +23 -0
- package/docs/reference/authoring-surface/type-workspacemachine.md +29 -0
- package/docs/reference/authoring-surface/type-workspaceof.md +13 -0
- package/docs/reference/authoring-surface/type-workspaceoptions.md +39 -0
- package/docs/reference/authoring-surface/type-workspacespec.md +37 -0
- package/docs/reference/authoring-surface/workspace.md +259 -0
- package/docs/reference/authoring-surface.md +197 -0
- package/docs/reference/cli.md +433 -0
- package/docs/reference/config.md +756 -0
- package/docs/reference/decisions/0001-operator-owned-generic-sandbox.md +113 -0
- package/docs/reference/decisions/0002-duplex-streaming-actor.md +110 -0
- package/docs/reference/decisions/0004-worktree-provisioning-model.md +124 -0
- package/docs/reference/decisions/0005-sandbox-pod-composition.md +230 -0
- package/docs/reference/decisions/0006-agent-control-surface.md +47 -0
- package/docs/reference/decisions/0007-durable-machine-state.md +109 -0
- package/docs/reference/decisions/0008-orchestrator-as-deployed-app.md +60 -0
- package/docs/reference/decisions/0009-cli-and-instance-interface.md +275 -0
- package/docs/reference/decisions/0010-bdd-acceptance-tests.md +160 -0
- package/docs/reference/decisions/0011-workflow-defined-events.md +141 -0
- package/docs/reference/decisions/0012-workspace-wrapper-machine.md +103 -0
- package/docs/reference/decisions/0013-the-agent-reaches-its-machine-through-a-container-it-cannot-read.md +157 -0
- package/docs/reference/decisions/0014-observation-is-open-run-state-is-not.md +95 -0
- package/docs/reference/decisions/0015-authoring-surface-absorbs-the-mechanism.md +265 -0
- package/docs/reference/decisions/0016-agent-turn-mechanics-are-internal.md +124 -0
- package/docs/reference/decisions/0017-pool-over-a-source-port.md +69 -0
- package/docs/reference/decisions/0018-instance-agents-are-definitions-jr2-assembles-the-harness.md +111 -0
- package/docs/reference/decisions/0019-one-converging-command-against-the-current-context.md +275 -0
- package/docs/reference/decisions/0020-private-ca-trust-is-a-named-concept.md +60 -0
- package/docs/reference/decisions/0021-workspace-continuity-is-a-lease-that-answers-back.md +100 -0
- package/docs/reference/decisions/0022-observation-is-a-level-triggered-feed.md +103 -0
- package/docs/reference/decisions/0023-the-harness-prints-the-conversation.md +176 -0
- package/docs/reference/decisions/0024-an-agents-turn-ends-with-the-state-that-asked-for-it.md +439 -0
- package/docs/reference/decisions/0025-cancel-ends-a-run-stop-parks-it.md +65 -0
- package/docs/reference/decisions/0026-a-turn-that-is-over-is-never-prompted.md +62 -0
- package/docs/reference/decisions/0027-the-harness-is-jr2s-own-server-flue-retires-the-wire-stays.md +237 -0
- package/docs/reference/decisions/0028-what-an-agent-may-do-to-the-workspace-is-part-of-its-definition.md +89 -0
- package/docs/reference/decisions/0029-a-menu-is-fixed-for-a-conversation-and-a-turn-is-told-its-allowed-picks.md +154 -0
- package/docs/reference/decisions/0030-a-snapshot-names-the-machine-it-was-written-under.md +100 -0
- package/docs/reference/decisions/0031-menu-only-agents-run-on-the-instance-harness.md +190 -0
- package/docs/reference/decisions/0032-the-console-unlocks-with-the-instance-token.md +61 -0
- package/docs/reference/decisions/0033-a-machine-declares-the-input-that-starts-it.md +127 -0
- package/docs/reference/decisions/0034-the-console-renders-with-preact-and-the-server-erases-the-types.md +57 -0
- package/docs/reference/decisions/0035-a-runaway-turn-is-ended-rerolled-once-then-a-fault.md +92 -0
- package/docs/reference/decisions/0036-a-turn-compacts-its-own-context-mid-flight.md +138 -0
- package/docs/reference/decisions/0037-an-instance-builds-its-sandbox-images-jr2-injects-the-harness.md +196 -0
- package/docs/reference/decisions/0038-jr2-up-builds-every-image-it-deploys.md +170 -0
- package/docs/reference/decisions/0039-image-garbage-collects-by-reachability.md +115 -0
- package/docs/reference/decisions/0041-a-build-the-host-already-holds-is-not-spent-again.md +64 -0
- package/docs/reference/decisions/0042-ready-is-not-routable-an-admission-retries-its-connection.md +142 -0
- package/docs/reference/decisions/0043-the-kit-is-tested-as-installed-a-local-registry-stands-in-for-npm.md +170 -0
- package/docs/reference/decisions/0044-kit-images-live-at-a-canonical-home-a-self-host-mirrors-it.md +85 -0
- package/docs/reference/decisions/0045-the-platform-joins-the-image-address-and-the-cluster-chooses-it.md +82 -0
- package/docs/reference/decisions/0046-a-converge-failure-carries-its-diagnosis.md +34 -0
- package/docs/reference/decisions/0047-the-git-ssh-key-source-is-the-users-choice.md +47 -0
- package/docs/reference/decisions/0048-the-orchestrator-boots-without-its-repos.md +36 -0
- package/docs/reference/decisions/0049-a-machine-carries-its-parts-and-composes-by-invoke.md +126 -0
- package/docs/reference/decisions/0050-what-code-names-config-declares-what-the-cli-names-discovery-finds.md +73 -0
- package/docs/reference/decisions/0051-a-repo-is-a-slot-on-the-workspace-and-a-cache-on-the-node.md +227 -0
- package/docs/reference/decisions/0052-a-sandbox-lands-where-an-ordinary-pod-lands-and-the-cache-follows-it.md +81 -0
- package/docs/reference/decisions/0053-a-fetch-inside-the-pod-asks-the-node-cache-and-the-cache-asks-the-remote.md +111 -0
- package/docs/reference/decisions/0054-the-kit-ships-machines-and-a-shipped-machine-leaves-its-parts-open.md +126 -0
- package/docs/reference/decisions/0055-a-release-is-a-pushed-tag-and-the-publish-guard-is-the-credential.md +107 -0
- package/docs/reference/decisions/0056-a-global-jr2-hands-off-to-the-instances-own-and-an-instance-has-one-kit-version.md +62 -0
- package/docs/reference/decisions/0057-a-turn-is-its-frame-its-dials-and-whether-it-continues.md +152 -0
- package/docs/reference/decisions/0058-a-harness-answers-the-orchestrator-alone.md +106 -0
- package/docs/reference/decisions/0059-a-harness-holds-stand-ins-and-the-custodian-holds-the-keys.md +384 -0
- package/docs/reference/decisions/0060-a-workspace-states-its-size-and-jr2-reserves-all-of-it.md +111 -0
- package/docs/reference/decisions/0061-a-memory-kill-ends-the-agents-processes-not-the-harness.md +83 -0
- package/docs/reference/decisions/0062-jr2-briefs-the-agent-on-its-seat-never-on-its-task.md +212 -0
- package/docs/reference/decisions/0063-the-orchestrator-watches-the-cluster.md +83 -0
- package/docs/reference/decisions/0064-a-workspace-waits-for-capacity.md +79 -0
- package/docs/reference/decisions/0065-a-persist-survives-a-crash-and-history-is-a-week.md +65 -0
- package/docs/reference/decisions/0066-a-model-tier-runs-a-real-model-and-a-change-to-what-it-reads-runs-the-tier.md +550 -0
- package/docs/reference/decisions/0067-an-effect-is-an-actor-slot-and-runs-at-most-once-unless-marked-idempotent.md +430 -0
- package/docs/reference/decisions/0068-an-effector-holds-an-effects-secrets-and-signs-its-wire.md +464 -0
- package/docs/reference/decisions/0069-the-orchestrator-is-a-kit-image-and-an-instance-publishes-its-assembly.md +180 -0
- package/docs/reference/decisions/0070-a-machine-lends-its-parts-to-a-chart.md +323 -0
- package/docs/reference/decisions/0071-a-run-starts-a-run-through-a-dispatch-effect.md +183 -0
- package/docs/reference/decisions/0072-the-docs-ship-inside-the-kit-and-an-instances-agents-md-indexes-them.md +144 -0
- package/docs/reference/decisions/0073-a-machine-fails-a-run-with-fail-and-an-uncaught-failure-ends-it-failed.md +122 -0
- package/docs/reference/decisions/0074-an-unrouted-mechanism-event-is-refused-at-load-and-faults-the-run.md +76 -0
- package/docs/reference/decisions.md +83 -0
- package/docs/reference/glossary.md +648 -0
- package/docs/reference/http-api.md +1348 -0
- package/docs/reference/machines.md +190 -0
- package/docs/reference/mechanism-events.md +72 -0
- package/docs/why.md +206 -0
- package/package.json +6 -4
- package/src/actor.ts +976 -61
- package/src/agent.ts +114 -38
- package/src/ambient.ts +4 -1
- package/src/assembly-checks.ts +347 -0
- package/src/assembly-tar.ts +494 -0
- package/src/assembly.ts +721 -0
- package/src/cause.ts +234 -0
- package/src/chart.ts +2182 -0
- package/src/charts.ts +748 -0
- package/src/config.ts +520 -162
- package/src/content-hash.ts +105 -0
- package/src/custodian.lua +6 -5
- package/src/custodian.ts +143 -39
- package/src/customize.ts +388 -55
- package/src/dispatch-effect.ts +945 -0
- package/src/door-compat.ts +217 -0
- package/src/durability.ts +11 -4
- package/src/effect-checks.ts +557 -0
- package/src/effect.ts +1202 -0
- package/src/effector-wire.ts +674 -0
- package/src/fail.ts +209 -0
- package/src/fingerprint.ts +4 -3
- package/src/gate.ts +8 -0
- package/src/git-clone.ts +65 -0
- package/src/git-push.ts +336 -0
- package/src/harness-client.ts +764 -48
- package/src/held-secrets.ts +82 -26
- package/src/holds.ts +156 -0
- package/src/http-effect.ts +1423 -0
- package/src/http.ts +916 -113
- package/src/images.ts +47 -21
- package/src/index.ts +62 -43
- package/src/instance.ts +505 -59
- package/src/internal.ts +84 -0
- package/src/kube-client.ts +333 -0
- package/src/machine-doc.ts +106 -6
- package/src/mechanism-routes.ts +168 -0
- package/src/model-keys.generated.ts +53 -0
- package/src/model-keys.ts +18 -58
- package/src/mount-digest.ts +53 -0
- package/src/names.ts +66 -8
- package/src/notices.ts +141 -0
- package/src/open-body.ts +1580 -0
- package/src/open-effect.ts +217 -0
- package/src/open.ts +59 -6
- package/src/parts.ts +709 -41
- package/src/party.ts +534 -0
- package/src/pick.ts +761 -0
- package/src/pool.ts +248 -37
- package/src/registration.ts +382 -19
- package/src/repo-fetch.ts +60 -85
- package/src/repos.ts +65 -75
- package/src/rest.ts +133 -0
- package/src/run-host.ts +2383 -115
- package/src/run-output.ts +38 -0
- package/src/sandbox-effect.ts +1126 -0
- package/src/sandbox-effector.ts +111 -0
- package/src/sandbox-kube.ts +1721 -0
- package/src/sandbox-watch.ts +260 -0
- package/src/secret-placement.ts +529 -0
- package/src/server.ts +284 -84
- package/src/setup.ts +819 -117
- package/src/size.ts +454 -0
- package/src/snapshot-store.ts +115 -20
- package/src/stub-harness.ts +49 -16
- package/src/sync-client.ts +54 -0
- package/src/timers.ts +191 -0
- package/src/tokens.ts +35 -4
- package/src/vocabulary.ts +135 -15
- package/src/wire.ts +238 -10
- package/src/workspace.ts +1195 -338
- package/src/sandbox-kubectl.ts +0 -1108
package/docs/reference/decisions/0059-a-harness-holds-stand-ins-and-the-custodian-holds-the-keys.md
ADDED
|
@@ -0,0 +1,384 @@
|
|
|
1
|
+
# A Harness holds stand-ins; the Custodian holds the keys
|
|
2
|
+
|
|
3
|
+
[ADR-0058](0058-a-harness-answers-the-orchestrator-alone.md) recorded an open gap: model API keys were in the Harness
|
|
4
|
+
container. `JR2_PROVIDER_API_KEY` and literal `harness.env` values (for example `ANTHROPIC_API_KEY`) came from the
|
|
5
|
+
`jr2-harness-env` Secret by `envFrom`. Working tools give the Agent code execution in that container, and every tool
|
|
6
|
+
child inherits the env. So the Agent could read a key and send it out of the pod. Dropping pi's catalog was not an
|
|
7
|
+
option: the user's real setup is Bedrock through a LiteLLM gateway, with a fixed `ANTHROPIC_API_KEY` and `anthropic/…`
|
|
8
|
+
models, and the catalog holds the limits, costs and thinking levels of those models.
|
|
9
|
+
|
|
10
|
+
The same problem had one solution already, for one credential. The Adapter held the Sandbox token in a container the
|
|
11
|
+
Agent executes nothing in ([ADR-0013](0013-the-agent-reaches-its-machine-through-a-container-it-cannot-read.md)). This
|
|
12
|
+
decision generalizes that container. It holds every credential the Harness in its pod uses, and it replaces the Adapter.
|
|
13
|
+
|
|
14
|
+
## Decision
|
|
15
|
+
|
|
16
|
+
- **A held secret is a credential the Harness uses and never holds.** The Harness container's env holds its
|
|
17
|
+
**Stand-in**, `jr2-held-<NAME>`. The Stand-in is deterministic and public. It means something only to the Custodian,
|
|
18
|
+
and only toward a host the secret is bound to. It is unique per name, it is a valid header and env value, and it never
|
|
19
|
+
contains `sk-ant-oat` (pi reads such a key as an Anthropic OAuth token). The word is not "placeholder", because that
|
|
20
|
+
is the word of an Open part.
|
|
21
|
+
- **The Custodian holds them.** It is a container in every Harness pod — every Sandbox and the Instance Harness — beside
|
|
22
|
+
the Harness, and it holds the held secrets its own pod names (below). The Agent executes no code in it, and
|
|
23
|
+
`shareProcessNamespace` stays off ([ADR-0005](0005-sandbox-pod-composition.md)). So the Agent cannot read its env, its
|
|
24
|
+
files or its `/proc`. Its engine is Envoy: the upstream distroless image, pinned by digest (`CUSTODIAN_IMAGE` in
|
|
25
|
+
`@jr2/orchestrator`). The pod's maker writes Envoy's whole configuration — the Orchestrator a Sandbox's at provision,
|
|
26
|
+
`jr2 up` the Instance Harness's — from the manifest `jr2 up` declares (see "Objects, per pod"). There is no admin
|
|
27
|
+
listener and no xDS, so nothing in the pod can change the configuration or read it back.
|
|
28
|
+
- **The Custodian replaces the Adapter.** The `@jr2/adapter` package and the `jr2-adapter` image are deleted. The
|
|
29
|
+
Sandbox token is a held secret of the kit's own: the Harness container holds
|
|
30
|
+
`JR2_SANDBOX_TOKEN=jr2-held-JR2_SANDBOX_TOKEN`, and the Custodian holds the token as a file from the `<sandbox>-token`
|
|
31
|
+
Secret (the Instance Harness: the `JR2_INSTANCE_HARNESS_TOKEN` key of `jr2-instance`). No other container mounts it.
|
|
32
|
+
- **The Harness reads its Menu itself.** There is no MCP. For each Submission, `menu.ts` reads
|
|
33
|
+
`GET /agents/<iid>/surface` and delivers a pick with `POST /agents/<iid>/events`, both to the Custodian on
|
|
34
|
+
`127.0.0.1:8081`, with the Stand-in as the bearer. `menu-tools.ts` gives pi each accepted event as a tool. The tool
|
|
35
|
+
names stay `mcp__jr2__<event>`, because shipped instructions and the printer depend on them; the name is no longer a
|
|
36
|
+
transport. `menu-tools.ts` adds the prefix, and `menu.ts` keeps the bare name, because an MCP client adds
|
|
37
|
+
`mcp__<server>__` itself. The Harness keeps what the Adapter did: a turn that is over is never prompted
|
|
38
|
+
([ADR-0026](0026-a-turn-that-is-over-is-never-prompted.md)), a receipt is prose
|
|
39
|
+
([ADR-0024](0024-an-agents-turn-ends-with-the-state-that-asked-for-it.md),
|
|
40
|
+
[ADR-0029](0029-a-menu-is-fixed-for-a-conversation-and-a-turn-is-told-its-allowed-picks.md)), the surface read retries
|
|
41
|
+
on the ladder with the `jr2.routability` line
|
|
42
|
+
([ADR-0042](0042-ready-is-not-routable-an-admission-retries-its-connection.md)), and a `poll` event refuses the turn
|
|
43
|
+
(`deferred` is retired, ADR-0006). The event's own JSON Schema is the tool's parameters, so the model sees the
|
|
44
|
+
contract the Orchestrator validates against.
|
|
45
|
+
- **The control listener forwards four routes and no other.** `127.0.0.1:8081`, plain HTTP, to the Orchestrator's
|
|
46
|
+
Service (`jr2-orchestrator.<ns>.svc:4000`, written into the pod's bootstrap by whoever makes the pod — see "Objects,
|
|
47
|
+
per pod"):
|
|
48
|
+
|
|
49
|
+
```
|
|
50
|
+
GET /agents/<iid>/surface → the same path the Menu
|
|
51
|
+
POST /agents/<iid>/events → the same path a pick
|
|
52
|
+
GET /agents/<iid>/deliveries/<id> → the same path an outstanding pick's answer (ADR-0024)
|
|
53
|
+
POST /fetch → /sandboxes/<this pod's Sandbox>/fetch the ask (ADR-0053)
|
|
54
|
+
anything else → 404, from the Custodian
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
A read — the surface, a delivery's poll — is retried on any reset; a pick and the ask only on a reset before they were
|
|
58
|
+
sent, so one the Orchestrator may have read is not sent twice. A poll holds up to about 25 s, far below the listener's
|
|
59
|
+
3600 s idle limit, so an outstanding pick never rides one long request (ADR-0002).
|
|
60
|
+
|
|
61
|
+
On each forwarded request, the Custodian swaps `Authorization: Bearer <Stand-in>` for the token and removes every
|
|
62
|
+
other credential header. A request without the Stand-in gets 403. The ask's path comes from the Custodian
|
|
63
|
+
(`JR2_SANDBOX` in its env, set per pod); on the Instance Harness there is none, and the ask gets 404. The route
|
|
64
|
+
allowlist is the second layer. The control stays in the Orchestrator: the token's scope, the live registration, and
|
|
65
|
+
the event's name and payload are judged there, as before (ADR-0013).
|
|
66
|
+
|
|
67
|
+
- **Model keys go the same way, over HTTPS.** When its pod holds a held secret, the Harness container also gets:
|
|
68
|
+
|
|
69
|
+
```sh
|
|
70
|
+
<NAME>=jr2-held-<NAME> one per held secret of the pod
|
|
71
|
+
HTTPS_PROXY / https_proxy=http://127.0.0.1:15001
|
|
72
|
+
NO_PROXY / no_proxy=localhost,127.0.0.1,::1
|
|
73
|
+
NODE_USE_ENV_PROXY=1
|
|
74
|
+
NODE_EXTRA_CA_CERTS=/etc/jr2/ca/extra.crt
|
|
75
|
+
SSL_CERT_FILE / REQUESTS_CA_BUNDLE / GIT_SSL_CAINFO=/etc/jr2/ca/bundle.crt
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
These ride `env`, after `harness.env` and before the wire's digest. In Kubernetes `env` wins over `envFrom`, so a
|
|
79
|
+
Stand-in overrides a same-named key of an `envFrom` Secret, even one that changes later. There is no `HTTP_PROXY`: the
|
|
80
|
+
Custodian carries TLS only, and plain HTTP (the Custodian's control listener, an in-cluster seed repository) goes
|
|
81
|
+
direct. `NO_PROXY` is loopback only, so an in-cluster gateway that a secret is bound to still goes through the
|
|
82
|
+
Custodian.
|
|
83
|
+
|
|
84
|
+
- **The egress listener: CONNECT only.** `127.0.0.1:15001`. The CONNECT target decides everything (lowercased, without a
|
|
85
|
+
trailing dot) — never the SNI, never the `Host` header. A target that a held secret is bound to goes to an internal
|
|
86
|
+
listener for that target alone. That listener ends TLS with a leaf `jr2 up` issued (ALPN `http/1.1` only) and, per
|
|
87
|
+
request:
|
|
88
|
+
- The method must be GET, HEAD, POST, PUT, PATCH, DELETE or OPTIONS, else 405. A server that answers TRACE echoes the
|
|
89
|
+
request, the value among it, back to the Agent. A header name with `_` gives 400: `x_api_key` is not `x-api-key` to
|
|
90
|
+
the strip, but some servers read it as one.
|
|
91
|
+
- `Host` must name the target, else 421. An encoded `/`, `.` or `\` in the path gives 400. A path outside the secret's
|
|
92
|
+
`paths` gives 403. It is checked after Envoy removes dot segments and reads `\` as `/` and `..;` as `..`, and the
|
|
93
|
+
upstream gets that path. So IIS and Tomcat cannot read a different path. A credential header sent twice gives 400.
|
|
94
|
+
- The swap: in each header the secret names, the Stand-in alone, or one auth scheme and the Stand-in
|
|
95
|
+
(`Bearer <Stand-in>`), becomes the value. Nothing else is rewritten.
|
|
96
|
+
- Every credential header that is not the product of a swap is removed. A request with no swap gets 403 and never
|
|
97
|
+
leaves the pod: `jr2 custodian: <target> needs the stand-in of <NAME> in <headers>`.
|
|
98
|
+
- Envoy opens its own TLS to the target, verifies the chain against `upstream.crt` and the name against the target's
|
|
99
|
+
SAN (DNS or IP), and adds no header either way (`server`, `x-envoy-*`, `x-request-id` and `x-forwarded-*` are off).
|
|
100
|
+
A failure gives 502 and a body that names the target and the reason. Redirects pass back unchanged.
|
|
101
|
+
- When the target gives no answer, the Custodian answers 502 with a body that names the target, says what jr2 knows
|
|
102
|
+
about the request, and ends with Envoy's reason. It replaces Envoy's own answer for ten Envoy response flags, in two
|
|
103
|
+
mappers. One reply can carry several flags, and Envoy uses the first mapper that matches, in order. The first mapper
|
|
104
|
+
is for `UF`, `UH` and `DF`, so a reply with one of them gets the first text whatever else it carries:
|
|
105
|
+
- `UF` (the connection to the target failed), `UH` (no healthy host) and `DF` (the DNS lookup failed) say that Envoy
|
|
106
|
+
did not send the request. The body is `jr2 custodian: <target> could not be reached: <details> <reason>`.
|
|
107
|
+
- `UC` (the target closed the connection), `DPE` (a downstream protocol error, which a body framing error can cause
|
|
108
|
+
after the headers went), `UPE` (an upstream protocol error), `UR` (the target reset the stream) and `LR` (Envoy
|
|
109
|
+
reset it) can come after the request went to the target. The body is
|
|
110
|
+
`jr2 custodian: <target> did not answer; the request may have reached it: <details> <reason>`. `UMSDR` (the
|
|
111
|
+
maximum stream duration) is in this mapper: it ends a stream that was sent. `URX` (the retry limit) is in it too:
|
|
112
|
+
Envoy sets it with the last attempt's own flag, so `UF,URX` gets the first text and `UC,URX` the second.
|
|
113
|
+
- Envoy does not set `URX` or `UMSDR` on these routes, because they set no retry policy and no
|
|
114
|
+
`max_stream_duration`. The two flags are in the mappers only to keep their union equal to the Orchestrator route's
|
|
115
|
+
list.
|
|
116
|
+
|
|
117
|
+
The second text is there because the first would be false after those flags. The target may have the request, so a
|
|
118
|
+
`curl -X POST` or a `gh api` call to a bound host can write twice when the Agent sends it again. "Could not be
|
|
119
|
+
reached" would tell the Agent that the first call went nowhere. The status stays 502 for both, the status of a
|
|
120
|
+
gateway whose target gave no valid answer. No code reads the words. The Harness's provider adapter reads the status,
|
|
121
|
+
and no model reads its own provider's 502. The Agent's own tools show the body to the model.
|
|
122
|
+
|
|
123
|
+
Without the mappers, Envoy answers each of the ten flags itself: 503 for most of them, and 502 for `UPE`. That body
|
|
124
|
+
names no target, so a caller can read it as the target's own answer. With them, every caller sees that the Custodian
|
|
125
|
+
gave the answer, and for which target. A reset after the target's answer started is not replaced: the caller sees
|
|
126
|
+
its stream end.
|
|
127
|
+
|
|
128
|
+
The tunnel (below) keeps one mapper for all ten flags, with the first text. Envoy answers a `CONNECT` with 200 only
|
|
129
|
+
after its TCP connection to the target is up, and no byte of the Agent's request goes before that 200. So a 502 on
|
|
130
|
+
the tunnel always comes before the request reached the target, and "could not be reached" is true for each flag.
|
|
131
|
+
|
|
132
|
+
The Orchestrator route is different ([ADR-0002](0002-duplex-streaming-actor.md)). Its mapper adds the header
|
|
133
|
+
`x-jr2-custodian: unreachable` and the flags, and the Harness reads them to decide whether a pick may be sent again
|
|
134
|
+
(`menu.ts`). It says "never reached" only when each flag of the reply is `UF`, `UH`, `DF` or `URX`, and "not known"
|
|
135
|
+
when any other flag is there.
|
|
136
|
+
|
|
137
|
+
- Bodies stream both ways. A route has no total timeout; an idle stream has an hour; a client that goes away resets
|
|
138
|
+
the upstream request.
|
|
139
|
+
|
|
140
|
+
Every other target is tunneled, byte for byte, after the dial guard (below).
|
|
141
|
+
|
|
142
|
+
- **The engine choices.** The values and every credential decision are in a Lua filter. Its program is fixed
|
|
143
|
+
(`program.lua`); each pod's own `custodian.lua` is a generated `HELD` table — names, headers, paths and targets, never
|
|
144
|
+
a value — that ends with `dofile` of the program, so the program runs after the table it reads. At start the program
|
|
145
|
+
reads each value from `/etc/jr2/custodian/values/<NAME>`, takes off one trailing `\n` or `\r\n`, and stops the
|
|
146
|
+
start-up with a line that names the secret when a value is missing, empty, or has a byte outside `0x20`–`0x7E`.
|
|
147
|
+
Envoy's `credential_injector` is not used: it is alpha, it breaks on a secret file that ends in a newline (1.39.1),
|
|
148
|
+
and it replaces a whole header, where the swap must see the Stand-in. Values never enter a ConfigMap. Envoy runs one
|
|
149
|
+
worker, with hot restart off, and flushes its access log twice a second.
|
|
150
|
+
- **The log is a contract.** One line per decision on stdout, with the prefix `jr2.custodian` and `key=value` pairs:
|
|
151
|
+
`ready secrets=… hosts=N` (at the first probe), `action=intercept target=… secret=… method=… status=…`,
|
|
152
|
+
`action=tunnel target=…`, `action=refuse target=… reason=…`. Never a header value, a body, a path, a query, a value or
|
|
153
|
+
a Stand-in.
|
|
154
|
+
- **The dial guard, in two halves.** A tunnel never reaches a loopback, link-local (the cloud metadata address included,
|
|
155
|
+
and AWS's `fd00:ec2::254`) or unspecified address, or an IPv4-mapped form of one; it gets 403 and a `reason=guard`
|
|
156
|
+
line. The script refuses a target that SPELLS one, and `localhost` or a `.localhost` name, before anything is
|
|
157
|
+
resolved. The dynamic forward proxy filter saves the address it resolved a name to (`save_upstream_address`), and an
|
|
158
|
+
RBAC filter after it refuses that address when it is in `GUARDED_RANGES` — so a name that resolves to one is refused
|
|
159
|
+
like the literal. The address judged is the address dialed: both come from the one DNS cache entry.
|
|
160
|
+
- **Health.** `0.0.0.0:15021`, `GET /healthz`. The kubelet cannot reach a loopback listener, and a distroless image has
|
|
161
|
+
no shell for an exec probe. The Custodian's readiness probe is on it. Pod Ready needs every container ready, and the
|
|
162
|
+
operator holds a Sandbox's Ready on pod Ready, so no Turn is admitted before the Custodian serves. The Instance
|
|
163
|
+
Harness's rollout wait covers it the same way. It is the one listener a peer can reach, where the ADR-0058
|
|
164
|
+
NetworkPolicy is not enforced. So it is outside Envoy's global connection limit, holds 16 connections at most, and
|
|
165
|
+
closes a connection idle for 5 seconds: a peer's idle connections cannot stop the Menu or the egress.
|
|
166
|
+
- **The config surface** (`jr2.config.ts`,
|
|
167
|
+
[ADR-0018](0018-instance-agents-are-definitions-jr2-assembles-the-harness.md)): the top-level `secrets` list, whose
|
|
168
|
+
entry is
|
|
169
|
+
`{ name; value | valueFrom.secretKeyRef; hosts?; headers?, default `authorization`, `x-api-key`, `x-goog-api-key`, `api-key`; paths? }`,
|
|
170
|
+
and `harness.catalog` (a catalog provider id → `{ baseUrl }`). The list serves both sides
|
|
171
|
+
([ADR-0068](0068-an-effector-holds-an-effects-secrets-and-signs-its-wire.md)). `harness.provider.apiKey` is an entry
|
|
172
|
+
too, implicitly: `JR2_PROVIDER_API_KEY`, bound to its `baseUrl`'s host. `loadConfig` checks the shape in both worlds.
|
|
173
|
+
- **Naming is placement: a pod gets the held secrets something in it names.** Placement is read, never declared
|
|
174
|
+
(`secretPlacementsOf`), and an entry's `hosts`, `headers` and `paths` are its binding — where the Custodian may put
|
|
175
|
+
the value — never its placement:
|
|
176
|
+
|
|
177
|
+
| Pod | The held secrets its Custodian holds |
|
|
178
|
+
| --------------------- | ------------------------------------------------------------------------------------------ |
|
|
179
|
+
| A Workspace's Sandbox | the names its `workspace()` lists under `secrets: [...]`, and the model keys of its Agents |
|
|
180
|
+
| The Instance Harness | the model keys of the `"none"` Agents it hosts, and nothing else |
|
|
181
|
+
|
|
182
|
+
A model key is claimed by the PROVIDER: an Agent names its model, and jr2 knows which keys that provider reads
|
|
183
|
+
(`MODEL_KEYS`; the custom provider's id for `JR2_PROVIDER_API_KEY`). An Agent's Turns run in the Workspace below which
|
|
184
|
+
it is declared, down to the next `workspace()`, unless it is `"none"`, whose Turns run at the Instance Harness
|
|
185
|
+
([ADR-0031](0031-menu-only-agents-run-on-the-instance-harness.md)). A `"none"` Agent cannot execute code, so the
|
|
186
|
+
Instance Harness loses nothing by holding only model keys. Listing a model key under a `workspace()`'s `secrets` is
|
|
187
|
+
allowed, and `jr2 up` reports it as redundant. A name a `workspace()` lists that the list does not hold is refused,
|
|
188
|
+
with the line to add. An entry nothing names — no `workspace()`, no provider, no Effect — goes nowhere, and `jr2 up`
|
|
189
|
+
warns. `jr2 up` reports per pod:
|
|
190
|
+
`Workspace of workflow "coding": Custodian holds ANTHROPIC_API_KEY (provider), JIRA_TOKEN`.
|
|
191
|
+
|
|
192
|
+
`jr2 up` alone checks the claimed entries (`heldSecretsOf`), host-side, because `.env` values are absent in-cluster.
|
|
193
|
+
An entry no pod claims is not checked as a held secret:
|
|
194
|
+
|
|
195
|
+
| # | Refused |
|
|
196
|
+
| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
197
|
+
| R1 | A known model key (`MODEL_KEYS`) in `harness.env`, or as a key of a Secret or ConfigMap `harness.envFrom` names. |
|
|
198
|
+
| R2 | A held-secret name that is `JR2_*` or a proxy or trust variable. |
|
|
199
|
+
| R3 | One name held twice, or held and set by `harness.env`. |
|
|
200
|
+
| R4 | Not exactly one of `value`/`valueFrom`; an empty value; a byte outside `0x20`–`0x7E`; leading or trailing space. |
|
|
201
|
+
| R5 | A claimed entry with no `hosts` where no default exists; a wildcard, userinfo, a non-`https` scheme, a bare-host path, a bad port; two secrets on one host in one header. |
|
|
202
|
+
| R6 | A `headers` entry that is not a lowercase token, or that frames or routes the request; a duplicate. |
|
|
203
|
+
| R7 | `provider.apiKey` with a non-`https` `baseUrl`; a non-`https` catalog `baseUrl`. |
|
|
204
|
+
| R8 | A catalog key that is not a known provider, or that is the custom provider's id. |
|
|
205
|
+
| R9 | Explicit `hosts` on a known model key that leave out where a carried Agent's provider sends its calls. |
|
|
206
|
+
| R10 | A `secretKeyRef` whose Secret or key does not exist. |
|
|
207
|
+
| R11 | While a secret is held: `harness.env` sets a proxy or trust variable. |
|
|
208
|
+
| R12 | A `paths` entry that does not start with `/`, or holds `..`, `;`, `\`, `%2e`, `%2f` or `%5c`. |
|
|
209
|
+
| R13 | A held value that also reaches the Harness: its Secret loaded by `harness.envFrom`, its key read by a `harness.env` `secretKeyRef`, or its literal set by `harness.env`. A kit Secret (`jr2-instance`, `jr2-held-*`, `jr2-instance-effector`, `jr2-effector-*`, `<sandbox>-effector`, `jr2-git-*`) named by either. |
|
|
210
|
+
|
|
211
|
+
Each message names the config path, never a value, and says what to write instead. A SigV4 or ADC credential in
|
|
212
|
+
`harness.env` is warned about, not refused. A known model key with no `hosts` is bound to the host its provider's
|
|
213
|
+
calls go to: the host of `harness.catalog.<id>.baseUrl` when set, else pi's catalog host. The key and the calls then
|
|
214
|
+
read one config and cannot drift. `MODEL_KEYS` is pi's data at the pin, written as plain data by `just model-keys` (a
|
|
215
|
+
script in `@jr2/harness`) at each pi bump, so the Orchestrator has no pi dependency. A test in `@jr2/harness` checks
|
|
216
|
+
the table against pi, and checks that the file is the script's exact text. So a pi bump that moves a provider, or a
|
|
217
|
+
hand edit of the file, fails the default gate until the recipe runs.
|
|
218
|
+
|
|
219
|
+
- **A catalog provider can move to a gateway.** pi has no env var for a catalog provider's base URL. It hands the
|
|
220
|
+
provider the model it is given. So `resolveModel` returns a copy of the catalog model with `baseUrl` replaced, for
|
|
221
|
+
every model of that provider. The catalog's other facts ride along. The model id sent upstream is the catalog id, so
|
|
222
|
+
the gateway must serve that name (a LiteLLM `model_name` alias).
|
|
223
|
+
- **Certificates.** `jr2 up` keeps a per-Instance CA in the `jr2-held-ca` Secret, which no pod mounts; it reads the CA
|
|
224
|
+
over the kube API. It issues one leaf per bound `host:port` into `jr2-held-tls` before a pod starts, so the CA's key
|
|
225
|
+
never enters a Harness pod. Keys are ECDSA P-256, written by a small DER writer in `@jr2/cli` over `node:crypto`: no
|
|
226
|
+
dependency, and no `openssl` on the host. `jr2 up` never rotates the CA; a human deletes the Secret to rotate it. A
|
|
227
|
+
leaf is reused while its SAN is the same, it chains to the current CA, and more than 30 days remain.
|
|
228
|
+
- **Trust.** The `jr2-ca` ConfigMap carries `ca.crt` (the user's `caBundle`), and while a secret is held: `extra.crt`
|
|
229
|
+
(`caBundle` + the jr2 CA, for `NODE_EXTRA_CA_CERTS`), `bundle.crt` (the roots + both, for the tools whose variable
|
|
230
|
+
replaces their store), and `upstream.crt` (the roots + `caBundle`, and never the jr2 CA, for the Custodian). "Roots"
|
|
231
|
+
are the Mozilla roots of the Node that runs `jr2 up`. The ConfigMap is applied server-side, because two copies of the
|
|
232
|
+
roots are more than client-side apply's annotation can hold.
|
|
233
|
+
- **Every Secret is applied server-side.** A client-side apply copies the whole object into its
|
|
234
|
+
`kubectl.kubernetes.io/last-applied-configuration` annotation, so the values of a Secret are then in plaintext for
|
|
235
|
+
each reader of its metadata. So `jr2 up` applies each Secret that it writes server-side: `jr2-instance`,
|
|
236
|
+
`jr2-harness-env`, `jr2-held-ca`, `jr2-held-tls`, `jr2-held-secrets-<pod>`, the Effector Secrets, the git Secrets, the
|
|
237
|
+
deploy key and the check pod's Secrets. The field manager is `jr2`, with `--force-conflicts`, because jr2 is the only
|
|
238
|
+
writer of these Secrets. A `stringData` goes as `data`, because a server-side apply owns the fields that it names, and
|
|
239
|
+
`stringData` is write-only. On a Secret that an older `jr2 up` applied client-side, the first server-side apply
|
|
240
|
+
migrates it: kubectl (1.26 and later) moves the fields of the client-side manager to `jr2`, so that apply removes the
|
|
241
|
+
annotation and each key that it does not name, and `jr2` is then the only manager (measured on kind with kubectl
|
|
242
|
+
1.35.9). When that migration fails, kubectl only writes a warning on stderr. So after each apply of Secrets, a
|
|
243
|
+
fallback removes the annotation (`kubectl annotate … last-applied-configuration-`). The removal of an annotation that
|
|
244
|
+
is not there changes nothing, so the fallback needs no record of what was removed. When the fallback fails, `jr2 up`
|
|
245
|
+
writes one warning that names the Secrets, and the converge goes on. The rule is in the apply seam
|
|
246
|
+
(`kubectlAdmin.apply`), so a Secret that a later change adds is applied in the same way. The values go on stdin, never
|
|
247
|
+
in argv. No ConfigMap holds a value.
|
|
248
|
+
- **Objects, per pod.** The held manifest, the Custodian's files and the literal values are each a pod's own, from the
|
|
249
|
+
one derivation that places Effect secrets. A pod is keyed `instance` for the Instance Harness, and `ws-<digest>` for a
|
|
250
|
+
Workspace, the digest of its registered workflow and the placement route to its `workspace()` (the pod key below) —
|
|
251
|
+
the route the walk reads statically and the run host reads off the live actor at provision, so both name one pod
|
|
252
|
+
alike. Nothing rendered per pod lands in a shared object: `jr2 up` declares, and the pod's maker renders. `jr2-held`
|
|
253
|
+
(always) holds `held-<pod>.json` for each pod (what it holds: names, bindings, headers, never a value) and
|
|
254
|
+
`program.lua` once per Instance, so it grows by one manifest per Workspace. A pod's two rendered files, Envoy's
|
|
255
|
+
`envoy.json` and its `custodian.lua`, are in a ConfigMap of that pod's own: the Orchestrator renders a Sandbox's at
|
|
256
|
+
provision, from the manifest it reads, into `<sandbox>-custodian`, applied after the CR and owned by it like the
|
|
257
|
+
`<sandbox>-token` Secret, so it goes with the pod (the Orchestrator's Role may create and patch a ConfigMap, and no
|
|
258
|
+
more); `jr2 up` renders the Instance Harness's into `jr2-held-custodian`, applied with its StatefulSet and deleted
|
|
259
|
+
with it. A pod's `custodian-config` volume projects its own ConfigMap and `program.lua` from `jr2-held`. The reason:
|
|
260
|
+
every Kubernetes object stops at 1 MiB, and client-side apply's annotation at 256 KiB; one copy of Envoy's files per
|
|
261
|
+
pod in `jr2-held` passed that limit at nine Workspaces, and a dedupe or a server-side apply only moves the ceiling.
|
|
262
|
+
`jr2-held-secrets-<pod>` holds one pod's literal values, labeled `jr2.dev/held-pod`, and `jr2 up` deletes the one a
|
|
263
|
+
pod no longer needs. `jr2-held-tls` (every bound target's leaf) and `jr2-held-ca` are the Instance's. The Orchestrator
|
|
264
|
+
mounts `jr2-held` at `/etc/jr2/held` and reads the manifest of the Workspace it provisions, per provision — the
|
|
265
|
+
`jr2-images` pattern ([ADR-0038](0038-jr2-up-builds-every-image-it-deploys.md)); a Workspace `jr2 up` never walked has
|
|
266
|
+
none, and its provision is refused, naming `jr2 up`. A pod key is the workflow and the Workspace's placement route:
|
|
267
|
+
the `actors` slot keys down to its `workspace()`, a wrapper's body skipped, where a plain Machine invoked inline adds
|
|
268
|
+
xstate's generated invoke src for that hop (`xstate.invoke.<index>.<state id>`), held so it can never equal a slot
|
|
269
|
+
key. `jr2 up` computes it from the state node, and the provision reads it off the live actor tree, so both derive one
|
|
270
|
+
key. A key reached by slot keys alone is what it was. A wrapper's body adds no hop, so a `workspace()` reached through
|
|
271
|
+
another `workspace()`'s body (`workspace(workspace(...))`, `workspace(pool(workspace(...)))`) has the outer one's key:
|
|
272
|
+
`jr2 up` refuses that Machine, with or without a secret, since the two pods' secrets could not be kept apart, and
|
|
273
|
+
names the fix (the inner `workspace()` under a key of a plain Machine's `actors`, which has a route of its own); every
|
|
274
|
+
other Workspace has a key of its own. A Workspace reached through a Machine object handed to `spawnChild`, which no
|
|
275
|
+
walk can see, fails at provision with the slot form. One function, `custodianComposition`, builds what a pod gains
|
|
276
|
+
from its pod's manifest; `kubeSandbox` and `instanceHarnessObjects` both call it, so the two pod shapes cannot drift.
|
|
277
|
+
The Instance Harness's pod template carries `jr2.dev/held-digest`, an HMAC over its pod's held-secret inputs keyed
|
|
278
|
+
with the signing key, so a held-secret edit rolls it at the next `jr2 up`. Its Custodian reads each value and its
|
|
279
|
+
files once at start, so the inputs are its two rendered files, the program, a literal's value, a `valueFrom`'s
|
|
280
|
+
reference, and a digest of the value in the user's Secret, keyed with the signing key: `jr2 up` reads that Secret to
|
|
281
|
+
check the key exists, and keeps the digest in place of the value, so a value edited there rolls the pod too. A live
|
|
282
|
+
Sandbox keeps what its Custodian read at start.
|
|
283
|
+
|
|
284
|
+
## Considered options
|
|
285
|
+
|
|
286
|
+
- **jr2's own proxy in Node, in a second container of the Adapter's image** (branch `held-secrets/node`). No new image
|
|
287
|
+
and no second language; jr2 then owns TLS-parsing code at the trust boundary, and needs its own tests for every edge
|
|
288
|
+
Envoy already handles. It keeps the Adapter.
|
|
289
|
+
- **Envoy in a container beside the Adapter** (branch `held-secrets/envoy-sidecar`). The same Envoy configuration, and
|
|
290
|
+
three containers per pod. Two containers hold two credentials, so a fault in one engine does not reach the other's
|
|
291
|
+
credential. This branch trades that split for one container fewer and one component fewer: the Custodian holds both.
|
|
292
|
+
- **The Orchestrator serves MCP, and Envoy forwards `/mcp/<iid>`.** Rejected. It reverses ADR-0013's "the Orchestrator
|
|
293
|
+
hosts no MCP", puts a transport and session handling back in the control plane, and keeps the MCP SDK on both ends —
|
|
294
|
+
to carry two plain HTTP calls. What ADR-0013 rejected about translation inside the Harness was the credential in that
|
|
295
|
+
container. The Custodian now holds it, so that reason is gone.
|
|
296
|
+
- **The Adapter relays model calls** (ADR-0058's first direction). Rejected: every provider needs its `baseUrl`
|
|
297
|
+
rewritten to the relay, and pi's catalog has no env var for a base URL, so the catalog could not follow.
|
|
298
|
+
- **Drop pi's catalog and name every model as a custom provider.** Rejected by the user: the catalog holds the model
|
|
299
|
+
facts the user would have to copy by hand.
|
|
300
|
+
- **iron-proxy.** It intercepts every host and cannot pass one through, and it makes leaves at runtime, so the CA's key
|
|
301
|
+
would be in the pod. The Sandbox Image is the user's, and its tools may not trust a jr2 CA.
|
|
302
|
+
- **mitmproxy, Squid.** mitmproxy streams SSE only if the addon asks, and its swap is code jr2 writes anyway; Squid
|
|
303
|
+
brings the GPL and a poor CVE record. Both would be a fourth image.
|
|
304
|
+
- **Envoy's `credential_injector`, or the value in a bootstrap Secret with `request_headers_to_add`.** Rejected: the
|
|
305
|
+
first is alpha and fails on a trailing newline; both replace a header without seeing the Stand-in, and the second
|
|
306
|
+
cannot hold a `secretKeyRef` value, which `jr2 up` never copies.
|
|
307
|
+
- **Intercept every host.** Rejected for the iron-proxy reason: a tool that does not trust the jr2 CA would then fail on
|
|
308
|
+
every host, not only on bound ones.
|
|
309
|
+
|
|
310
|
+
## Consequences
|
|
311
|
+
|
|
312
|
+
- **What this closes, and what it does not.** The Agent cannot take a model key or the Sandbox token out of the pod. It
|
|
313
|
+
can still **spend** a key through the Custodian while its pod lives, on any path `paths` allows. A key with admin
|
|
314
|
+
rights can mint a new key, and a minted key is one the Agent can take away: use a key with no admin rights, and narrow
|
|
315
|
+
`paths`. The User Container shares the pod's loopback, so a human in it can spend a key the same way. That human
|
|
316
|
+
cannot read one either.
|
|
317
|
+
- **Response echo.** Responses pass unchanged. An upstream that echoes the full credential in a response hands it to the
|
|
318
|
+
Agent. Providers mask keys in errors; jr2 does not scrub responses.
|
|
319
|
+
- **Bypass.** A client that ignores `HTTPS_PROXY` goes direct and sends the Stand-in, which is worth nothing. It is not
|
|
320
|
+
blocked: egress control stays the cluster's decision (ADR-0058's open egress gap). The same holds for the control
|
|
321
|
+
plane: a caller that skips the Custodian and sends the Stand-in to the Orchestrator gets 401 on every route
|
|
322
|
+
(`auth.test.ts`).
|
|
323
|
+
- **Clients that do not trust the jr2 CA** (Java, a pinned client) fail TLS to a bound host. They fail closed and send
|
|
324
|
+
nothing. Unbound hosts still work for them.
|
|
325
|
+
- **`SSL_CERT_FILE` replaces an OpenSSL tool's store.** A private CA baked into the Sandbox Image, and not put in
|
|
326
|
+
`caBundle`, stops being trusted by curl, git and Python in a pod with a held secret. The fix is `caBundle`. The gain:
|
|
327
|
+
`caBundle` now reaches curl, git and Python, which [ADR-0020](0020-private-ca-trust-is-a-named-concept.md) recorded as
|
|
328
|
+
missing.
|
|
329
|
+
- **Signing credentials cannot be held.** A SigV4 or ADC credential signs the request with the secret, and a Stand-in
|
|
330
|
+
cannot be swapped into a signature. `jr2 up` warns. The fix is the Effector
|
|
331
|
+
([ADR-0068](0068-an-effector-holds-an-effects-secrets-and-signs-its-wire.md)): an Effect secret is the real value, in
|
|
332
|
+
a container the Agent executes nothing in. Its `http` kind sends a bearer, a Basic pair or a header the author writes,
|
|
333
|
+
and the credentialed `command`
|
|
334
|
+
([ADR-0067](0067-an-effect-is-an-actor-slot-and-runs-at-most-once-unless-marked-idempotent.md)) runs the signing tool
|
|
335
|
+
there (`terraform`, `aws`, `gcloud`), its child confined with Landlock from the Effector's key and every secret it
|
|
336
|
+
does not name. `ANTHROPIC_OAUTH_TOKEN` is held like a key, but pi sends a Stand-in as `x-api-key`, which an OAuth
|
|
337
|
+
token cannot ride; hold it as `ANTHROPIC_AUTH_TOKEN` (`Authorization: Bearer`) instead.
|
|
338
|
+
- **Rotation of a value** reaches a live Sandbox only when its pod is replaced.
|
|
339
|
+
- **The walk reads definitions, not Turns.** A pod's model keys come from the models its Agents' definitions name. A
|
|
340
|
+
Turn whose invocation overrides the model with another provider's (`AgentTurnInput.model`) finds no key for that
|
|
341
|
+
provider in its pod: the Harness container gets a Stand-in only for the names in its own pod's manifest, so that
|
|
342
|
+
provider's key variable is not set at all. The call then has no key (a catalog provider) or sends the custom
|
|
343
|
+
provider's placeholder `unused`, and it fails. This holds in a Workspace and in the Instance Harness alike. The fix is
|
|
344
|
+
a definition that names the provider; in a Workspace, the key under the `workspace()`'s `secrets` also works. The
|
|
345
|
+
Instance Harness has no such list: its pod holds only the model keys of the `"none"` Agents it hosts.
|
|
346
|
+
- **TLS-only bindings.** A key for a plain-HTTP endpoint is refused (R7). A LAN `llama-server --api-key` needs TLS and
|
|
347
|
+
its CA in `caBundle`, or no key.
|
|
348
|
+
- **No proxy chaining.** The Custodian cannot forward to a corporate proxy. R11 says so rather than fail silently.
|
|
349
|
+
- **Two Harness env shapes.** With no held secret in it, a pod gets the Custodian's address and the token's Stand-in and
|
|
350
|
+
no proxy, and its HTTPS goes direct. The Custodian runs in both shapes, because it carries the Menu.
|
|
351
|
+
- **The dial guard adds no confinement while egress is open.** The Agent can dial those addresses itself. The guard
|
|
352
|
+
matters when a cluster closes egress to everything but the Custodian: then the Custodian is not a way around it.
|
|
353
|
+
- **One container holds two kinds of credential.** A fault in Envoy's TLS or HTTP parsing is a fault in the container
|
|
354
|
+
that holds both the Sandbox token and the model keys. The branch with a separate Adapter keeps them apart.
|
|
355
|
+
- **The Harness container now speaks the Orchestrator's two agent routes.** The Agent could always make those calls (the
|
|
356
|
+
Adapter would forward any MCP call it asked for); now the Harness makes them itself. Nothing is judged in the pod.
|
|
357
|
+
- **`jr2-upload-pack` carries the Stand-in as a constant.** The User Container gets no env (ADR-0005), and asks like any
|
|
358
|
+
other seat. A test pins the Go literal to `standIn("JR2_SANDBOX_TOKEN")`.
|
|
359
|
+
- **The Custodian's image is a Pinned image.** It is deployed and never built, which
|
|
360
|
+
[ADR-0038](0038-jr2-up-builds-every-image-it-deploys.md) now states. `kitRegistry` re-homes it and `jr2 kit push`
|
|
361
|
+
mirrors it ([ADR-0044](0044-kit-images-live-at-a-canonical-home-a-self-host-mirrors-it.md)). The image sweep never
|
|
362
|
+
takes it, because jr2 never stamped it ([ADR-0039](0039-image-garbage-collects-by-reachability.md)). A kind node pulls
|
|
363
|
+
it from its registry, so the `@kind` tier needs network to `docker.io`. `just kit-push` does not seed it.
|
|
364
|
+
- **Where the Custodian suite runs.** `just custodian-test`: node's test runner against the pinned image under docker,
|
|
365
|
+
on the host network with free ports, with local HTTPS upstreams under their own CA. It is opt-in, like `@kind`,
|
|
366
|
+
because it needs docker ([ADR-0010](0010-bdd-acceptance-tests.md)), and the release job runs it on every tag
|
|
367
|
+
([ADR-0055](0055-a-release-is-a-pushed-tag-and-the-publish-guard-is-the-credential.md)): the image moves only with its
|
|
368
|
+
pin, but the configuration jr2 renders moves with any release. About 35 seconds. It checks the swap, the strip, 403
|
|
369
|
+
with no Stand-in, 405 on a method, 400 on an underscore header, a header `Connection` names, two `Host` headers, idle
|
|
370
|
+
connections on the health listener, 421, 400 and 403 on paths, the tunnel, the dial guard by every spelling and by
|
|
371
|
+
resolved name, 502 on an untrusted upstream, SSE timing, abort, a 302, the start-up refusal, the log, and the
|
|
372
|
+
Anthropic SDK through `HTTPS_PROXY`. The first SSE event arrives through the Custodian about half a millisecond later
|
|
373
|
+
than direct.
|
|
374
|
+
- **The `@kind` tier's model wants a key.** Its fake provider serves HTTPS under the tier's CA and refuses a request
|
|
375
|
+
without the key, which is `harness.provider.apiKey` — so every turn in the tier crosses a Custodian, in both
|
|
376
|
+
placements. A Rule proves the key is in no env and no file of either Harness container.
|
|
377
|
+
- **The provider preflight crosses a Custodian.** A provider's key is held. So `jr2 up` checks the provider from a pod
|
|
378
|
+
with a Custodian that holds its keys, and the stock Harness's `provider-preflight.ts`
|
|
379
|
+
([ADR-0019](0019-one-converging-command-against-the-current-context.md)). The check pod reads its own copies of the
|
|
380
|
+
held objects. The check runs before `jr2 up` writes an object that the Instance or the operator reads. A literal key
|
|
381
|
+
is in the check pod's own Secret while the pod lives. A wrong or missing key, no egress to the bound host, and a
|
|
382
|
+
gateway that does not answer fail the converge, not a Turn.
|
|
383
|
+
- **The CA has no name constraints.** Constraints would force a new CA whenever the bound hosts change, and that breaks
|
|
384
|
+
a live pod whose Custodian restarts.
|
package/docs/reference/decisions/0060-a-workspace-states-its-size-and-jr2-reserves-all-of-it.md
ADDED
|
@@ -0,0 +1,111 @@
|
|
|
1
|
+
# A Workspace states its Size, and jr2 reserves all of it
|
|
2
|
+
|
|
3
|
+
No jr2 pod asked Kubernetes for anything. `Sandbox.spec.resources` existed and nothing filled it; the Custodian, the
|
|
4
|
+
Instance Harness and the Orchestrator set nothing either. Every pod was BestEffort, so the scheduler packed Sandboxes
|
|
5
|
+
without limit, the cluster autoscaler saw no demand, and a full cluster showed up as OOMKill and eviction instead of
|
|
6
|
+
Pending — an eviction takes `/work` with it (ADR-0021). The scaling review (2026-09-27) and a lab of real workloads on
|
|
7
|
+
the home cluster (2026-09-28: Playwright, `tsc` + vitest on hono, `cargo build` of ripgrep; one run each) measured what
|
|
8
|
+
a Sandbox uses. The owner plans to move Workspaces onto Agent Substrate later, so the shape is chosen to move there
|
|
9
|
+
unchanged.
|
|
10
|
+
|
|
11
|
+
## Decision
|
|
12
|
+
|
|
13
|
+
- **A Workspace states one Size: a ceiling for its whole Sandbox, cpu and memory, nothing else.** The shape is Agent
|
|
14
|
+
Substrate's `ActorTemplate.resources`: a Kubernetes `ResourceRequirements` of which only `limits.cpu` and
|
|
15
|
+
`limits.memory` are accepted; `requests` and any other key are refused where they are read.
|
|
16
|
+
`workspace(body, { resources: { limits: { memory: "3Gi", cpu: "2" } } })`.
|
|
17
|
+
- **The Size is a fact of the Machine**, like its Sandbox Image (ADR-0037, ADR-0049): the author knows the toolchain,
|
|
18
|
+
the runner, the worker count. It is never per run. The resolution chain, most specific first:
|
|
19
|
+
`customize(machine, { resources })` (the composer, whose Repo may be bigger than the author's) → the `workspace()` →
|
|
20
|
+
`sandbox.resources` in `jr2.config.ts` (the Instance's default, applied **only** to a Workspace that states none —
|
|
21
|
+
never an override of a stated Size, so each fact has one override path) → the kit default.
|
|
22
|
+
- **jr2 reserves the whole Size.** Memory request = limit and cpu request = limit: the Sandbox pod is Guaranteed. Memory
|
|
23
|
+
pressure never picks a Workspace, and placement accounts the way Substrate's does. Density is paid for here and
|
|
24
|
+
recovered later by suspend (the scaling review's R15), not by overcommit.
|
|
25
|
+
- **The CPU limit is enforced.** Nothing reads a CPU request (the cgroup shows `cpu.max = max`), while Node's
|
|
26
|
+
`availableParallelism`, Go 1.25, Rust and cargo, the JVM, .NET and every Substrate runtime read the limit. Throttling
|
|
27
|
+
is the accepted cost.
|
|
28
|
+
- **Tools that read neither the limit nor affinity get hints**, from the Downward API (`limits.cpu`, divisor 1):
|
|
29
|
+
`JR2_CPUS`, `OMP_NUM_THREADS` (which also makes `nproc` answer N), `PYTHON_CPU_COUNT`, `GOMAXPROCS`. `os.cpus()` still
|
|
30
|
+
reports the node, and Playwright sizes its workers from it alone, so a Machine passes `--workers=$JR2_CPUS`; the Agent
|
|
31
|
+
is told its CPU count by the Briefing ([ADR-0062](0062-jr2-briefs-the-agent-on-its-seat-never-on-its-task.md)), never
|
|
32
|
+
by the Frame.
|
|
33
|
+
- **jr2 splits the Size inside the pod.** The Custodian takes a fixed kit slice (64Mi, 50m). The User Container takes
|
|
34
|
+
its split if the Machine states one — `user` widens from a string to
|
|
35
|
+
`{ image, resources?: { limits: { memory?, cpu? } } }` (ADR-0005 foresaw the widening) — and otherwise shares the pod
|
|
36
|
+
budget with no limit of its own. The Sandbox Effector, in a pod that has one
|
|
37
|
+
([ADR-0068](0068-an-effector-holds-an-effects-secrets-and-signs-its-wire.md)), has no split and no limit of its own
|
|
38
|
+
and shares the pod budget the same way. The Harness container gets the rest as its own limit, so an OOM stays per
|
|
39
|
+
container where it can. Pod-level resources (KEP-2837) hold the whole-pod ceiling. A server with the PodLevelResources
|
|
40
|
+
gate off drops them without a word, so `jr2 up` dry-runs a pod that states them and refuses the cluster when they are
|
|
41
|
+
gone. `jr2 up` refuses a split that leaves the Harness below its floor (256Mi) and names the line.
|
|
42
|
+
- **Kit numbers.** Default Size: memory 2Gi, cpu 1. Node gives each process a V8 heap of about 55% of its container
|
|
43
|
+
limit, and hono's `tsc` aborted with a heap error below about 1.1Gi, so 1Gi fails a common task. 2Gi covers `tsc` +
|
|
44
|
+
vitest, `cargo build` and Playwright at 2 workers; Playwright at 4 workers and large monorepos state a Size. No
|
|
45
|
+
`NODE_OPTIONS` heap raise: a V8 abort is a tool error the Agent reads, while a bigger heap turns it into a kernel
|
|
46
|
+
kill.
|
|
47
|
+
- **`/dev/shm` is a memory-backed emptyDir in the Harness container only**, `sizeLimit` = 25% of the Harness share
|
|
48
|
+
(minimum 64Mi). It reserves nothing extra (shm is charged inside the Harness limit), and a full shm is ENOSPC or
|
|
49
|
+
SIGBUS — a tool error, never an OOM (lab). The Harness clears it at start: shm files outlive a container restart and
|
|
50
|
+
stay charged to the pod, which crash-looped the restarted container in the lab. There is no `shm` field; a Machine
|
|
51
|
+
that needs more raises its Size.
|
|
52
|
+
- **`/work` disk is a kit-fixed ephemeral-storage request of 10Gi on the Harness container, with no cap.** The request
|
|
53
|
+
places the pod and ranks it last under node disk pressure; no `sizeLimit` and no limit, because passing one evicts the
|
|
54
|
+
pod and loses `/work` (T12). Disk is not part of the Size: Substrate's template size is cpu and memory only.
|
|
55
|
+
- **Priority.** `jr2 up` creates two cluster-scoped PriorityClasses beside the CRDs: `jr2-control` (100000,
|
|
56
|
+
PreemptLowerPriority) for the Orchestrator, the operator and the Repo cache agent — small, fixed and needed by every
|
|
57
|
+
Sandbox — and the Instance Effector ([ADR-0068](0068-an-effector-holds-an-effects-secrets-and-signs-its-wire.md)),
|
|
58
|
+
small and fixed; `jr2-sandbox` (1000, preemptionPolicy `Never`) for Sandboxes and the Instance Harness — a pod of
|
|
59
|
+
ordinary priority cannot preempt a live Workspace, and a waiting Sandbox evicts nobody. Both stay far below the system
|
|
60
|
+
classes. A cluster with its own scheme names its classes in `priorityClasses: { control?, sandbox? }`.
|
|
61
|
+
- **Voluntary disruption.** A Sandbox pod and the Instance Harness pod carry
|
|
62
|
+
`cluster-autoscaler.kubernetes.io/safe-to-evict: "false"` and `karpenter.sh/do-not-disrupt: "true"`: moving either
|
|
63
|
+
loses work (`/work`; live conversations). The Orchestrator, the operator, the Repo cache agent and the Instance
|
|
64
|
+
Effector carry neither — a restarted Orchestrator re-attaches (ADR-0007), and a stopping Effector finishes its Effects
|
|
65
|
+
first. No PodDisruptionBudget: `maxUnavailable: 0` blocks node upgrades without end. A drain ends in `workspace.lost`,
|
|
66
|
+
and the body's policy decides (ADR-0021).
|
|
67
|
+
- **jr2's own pods are kit-sized, with no config key**: memory request = limit, a CPU request, no CPU limit (a
|
|
68
|
+
provisioning burst needs CPU, and no Substrate move applies to them). Orchestrator 500m / 1Gi. Instance Harness 250m /
|
|
69
|
+
1Gi (it frees every conversation no live run holds — ADR-0031). Operator 50m / 256Mi (it caches every Pod in the
|
|
70
|
+
cluster). Instance Effector 50m / 128Mi. Repo cache agent 50m, memory request 128Mi and limit 1Gi — the one exception,
|
|
71
|
+
since `git index-pack` spikes and its OOM costs only a refetch.
|
|
72
|
+
|
|
73
|
+
## Considered options
|
|
74
|
+
|
|
75
|
+
- **Sizes as named classes in `jr2.config.ts`, which the Machine names** (`workspace(body, { class: "browser" })`, the
|
|
76
|
+
config maps the class to resources). Rejected: it breaks packaging. Two packages each name `browser` and mean
|
|
77
|
+
different sizes, and the consumer must supply numbers only the package's author knew. ADR-0052's named class stays
|
|
78
|
+
placement only — node labels are the cluster's facts; a memory need is the Machine's.
|
|
79
|
+
- **Raw `ResourceRequirements`, all of it.** Rejected: the author could set a request below the limit, leave out the
|
|
80
|
+
memory limit, or reach fields Substrate refuses, and a Machine would depend on something the move drops.
|
|
81
|
+
- **A jr2 shape** (`{ memory, memoryLimit?, cpu }`). Rejected for the same reason in the other direction: one more
|
|
82
|
+
translation at the move, for no gain the Substrate subset does not give.
|
|
83
|
+
- **Burstable: reserve the working peak, allow bursts to the limit.** About twice the density. Rejected: with no
|
|
84
|
+
suspend, several Workspaces bursting at once is node pressure, and pressure evicts a whole pod with its `/work`.
|
|
85
|
+
- **Reserve CPU, never throttle.** Chosen first, then reversed: no mechanism tells a tool its CPU request (the lab tried
|
|
86
|
+
affinity, CPU Manager, LXCFS, gVisor, Kata and `LD_PRELOAD` shims; all but affinity need a limit, and affinity misses
|
|
87
|
+
`os.cpus()` and lets pods pin the same cores), while every container-aware runtime reads the limit.
|
|
88
|
+
- **An `shm` field, or a `/work` disk field, on the Size.** Deferred: Substrate has neither. A jr2 field beside the Size
|
|
89
|
+
can come when a real Machine needs one.
|
|
90
|
+
- **`/dev/shm` without a `sizeLimit`, or the container's 64Mi default.** Rejected: unbounded, a full shm OOM-kills the
|
|
91
|
+
container and poisons its restart; 64Mi crashed Chromium renderers when Playwright's `--disable-dev-shm-usage` was
|
|
92
|
+
off.
|
|
93
|
+
|
|
94
|
+
## Consequences
|
|
95
|
+
|
|
96
|
+
- ADR-0005 is rewritten in place: containers now carry limits from the Size, the User Container may take a split, and
|
|
97
|
+
`/dev/shm` joins the Harness container's mounts.
|
|
98
|
+
- ADR-0052 gains `sandbox.resources` beside `nodeSelector` and `tolerations`; its reserved named class stays placement
|
|
99
|
+
only, so its `node:` name stays.
|
|
100
|
+
- An OOM in the Harness container still kills the Harness with the Agent's processes (cgroup v2 group kill). Containing
|
|
101
|
+
it is ADR-0061.
|
|
102
|
+
- A Workspace parked on a Gate holds its whole Size. That is the density problem R15 exists for.
|
|
103
|
+
- On a move to Substrate, a Workspace with a User Container split needs a microVM pool: Substrate accepts per-container
|
|
104
|
+
limits on microVM runtimes only, and gVisor refuses them.
|
|
105
|
+
- `jr2 up` warns — never refuses — when no current Sandbox node could hold a Workspace's Size (the ADR-0052 stance: the
|
|
106
|
+
node set moves).
|
|
107
|
+
- GLOSSARY.md gains **Size**.
|
|
108
|
+
- The kit-sized control pods ask a node for about 900m a scenario before any Sandbox, so the `@kind` tier on a 4-vCPU
|
|
109
|
+
hosted runner runs two workers, not the profile's four (`just e2e-kind 2` in the release job). v0.4.0's release run
|
|
110
|
+
measured the alternative: four workers left every Sandbox Pending on `Insufficient cpu`. The requests are the kit's
|
|
111
|
+
and are not lowered for a runner.
|
package/docs/reference/decisions/0061-a-memory-kill-ends-the-agents-processes-not-the-harness.md
ADDED
|
@@ -0,0 +1,83 @@
|
|
|
1
|
+
# A memory kill ends the Agent's processes, not the Harness
|
|
2
|
+
|
|
3
|
+
The Agent runs its programs (Playwright and its Chromium, `tsc`, `cargo`) through the `bash` Working tool, so they are
|
|
4
|
+
children of the Harness, in the Harness container's one cgroup (ADR-0005, ADR-0037). On cgroup v2 the kubelet sets
|
|
5
|
+
`memory.oom.group`, so when those programs pass the container's limit (ADR-0060) the kernel kills the whole container:
|
|
6
|
+
the Harness dies with them, its conversation with it, and the Turn faults as "conversation lost" (lab 2026-09-28; chaos
|
|
7
|
+
test T11). Worse, the bash tool that pi-agent-core 0.82.1 shipped reported a command killed by a signal as success
|
|
8
|
+
(`exitCode: code ?? 0`), so when a program was killed the Agent saw its output stop partway and read it as a pass.
|
|
9
|
+
|
|
10
|
+
**The Agent's processes** are every process in the Harness container except the Harness and its init: the tree of the
|
|
11
|
+
running `bash` call, anything the Agent left running from earlier calls (a dev server, a watcher), and a human's
|
|
12
|
+
`kubectl exec` session.
|
|
13
|
+
|
|
14
|
+
## Decision
|
|
15
|
+
|
|
16
|
+
Layers, each a backstop for the one before:
|
|
17
|
+
|
|
18
|
+
1. **A process killed by a signal is an error.** jr2's bash tool reads the shell's exit: a signal death, of the command
|
|
19
|
+
or of the shell, reports the signal and never exit 0. A Harness conformance test holds it (ADR-0027).
|
|
20
|
+
2. **`tini` is PID 1 in the Harness container**, shipped on `/opt/jr2`, so killed trees leave no zombies. Not
|
|
21
|
+
`shareProcessNamespace` — ADR-0005 keeps it off for good.
|
|
22
|
+
3. **The `bash` Working tool starts each command with `oom_score_adj=1000`**, inherited by its children. It needs no
|
|
23
|
+
privilege, and on a node whose owner set the kubelet's `singleProcessOOMKill: true` the kernel then picks the Agent's
|
|
24
|
+
process, not the Harness.
|
|
25
|
+
4. **The Harness guards its own cgroup.** Every 10–20 ms it reads anon + shmem from `memory.stat` against `memory.max`
|
|
26
|
+
(never `memory.current`, which counts reclaimable page cache). Near the limit it kills all of the Agent's processes,
|
|
27
|
+
checks usage again, clears `/dev/shm`, and the running `bash` call returns
|
|
28
|
+
`killed: memory limit (peak X of Y); use fewer workers or a larger Size`. The Agent hears it in the same Turn and
|
|
29
|
+
conversation, and can retry smaller. The guard polls from the Harness's start, not only during a call. A process that
|
|
30
|
+
an earlier Turn left running (a dev server, a watcher) can pass the limit while no Submission runs. For example, the
|
|
31
|
+
workflow waits at a human's Gate. Then no call returns the text. The Harness keeps every kill for the Orchestrator to
|
|
32
|
+
read, and a read removes nothing. The next Agent that works in that Workspace is told (ADR-0062).
|
|
33
|
+
5. **The kernel's group kill is the last backstop.** The Orchestrator reads `lastState.terminated.reason: OOMKilled` and
|
|
34
|
+
the fault reason starts with the fixed prefix `memory limit`, not `Turn lost` (any other restart, ADR-0021) or
|
|
35
|
+
"conversation lost" (a 404). The fault reaches the Machine (ADR-0016); the next Agent in that Workspace is told
|
|
36
|
+
through the Briefing ([ADR-0062](0062-jr2-briefs-the-agent-on-its-seat-never-on-its-task.md)), never through the
|
|
37
|
+
Frame (ADR-0057). The kit's Machines pass the reason into their retry Turn as the example for authors. A kernel kill
|
|
38
|
+
while no Submission runs faults no Turn, because the restarted Harness rebuilds its conversations. The operator
|
|
39
|
+
publishes the container's last end on the Sandbox all the same, so before each admission into the Workspace the
|
|
40
|
+
Orchestrator reads it from the Sandbox watch, and a last end `OOMKilled` that no notice told yet tells the next Agent
|
|
41
|
+
(ADR-0062).
|
|
42
|
+
|
|
43
|
+
The guard is best-effort and says so: a fast enough spike beats any poll. The lab won 35/35 on arm64 at 50 ms, 6/6 on
|
|
44
|
+
amd64 at 10 ms (0.8% of a core), and only half on amd64 at 50 ms; it fired at 90–96% of the limit. Layer 5 covers what
|
|
45
|
+
it misses.
|
|
46
|
+
|
|
47
|
+
**The planned successor is a child cgroup for the Agent's processes** (`memory.max` below the container's, its own
|
|
48
|
+
`oom.group`): exact, no race, the Harness always survives (lab: 3/3 on kind). It needs a writable cgroupfs, which today
|
|
49
|
+
only a node owner grants (containerd ≥ 2.1 `cgroup_writable` behind a RuntimeClass, not offered on managed clusters).
|
|
50
|
+
KEP-5474 adds `securityContext.cgroupOptions.mountMode: Writable` so the pod asks for it itself; it is alpha, targeted
|
|
51
|
+
at v1.38. When it is enabled by default, jr2 sets the field on the Harness container and the child cgroup replaces layer
|
|
52
|
+
4's polling.
|
|
53
|
+
|
|
54
|
+
## Considered options
|
|
55
|
+
|
|
56
|
+
- **Accept the group kill.** Rejected as the whole answer: every memory kill becomes a Turn fault, and the Agent never
|
|
57
|
+
learns why, so a retry repeats it.
|
|
58
|
+
- **The Harness in its own container, with an exec server in the Sandbox Image's container running the Agent's
|
|
59
|
+
processes.** A real cgroup edge, but it reverses "Working tools execute in the Harness container" (ADR-0037) for a
|
|
60
|
+
problem layers 4 and the successor solve inside it.
|
|
61
|
+
- **Adopt the child cgroup now, used when detected.** Rejected: a path almost no cluster runs is a path nobody tests,
|
|
62
|
+
and the guard must be solid on its own anyway.
|
|
63
|
+
- **MemoryQoS `memory.high`.** Rejected: the lab saw a stall of over 9 minutes with no kill and `kubectl exec` hung.
|
|
64
|
+
- **earlyoom, systemd-oomd, oomd.** Rejected: they read the host's `/proc/meminfo`, need systemd, or need PSI triggers,
|
|
65
|
+
which a read-only cgroupfs refuses. The guard borrows their idea.
|
|
66
|
+
- **Kill only the running call's process group.** Rejected: `setsid` children escape it (lab), and a dev server left
|
|
67
|
+
from an earlier call may hold the memory.
|
|
68
|
+
- **Raise Node's heap for the Agent's processes** (`NODE_OPTIONS`). Rejected in ADR-0060: it turns a readable V8 abort
|
|
69
|
+
into a kernel kill.
|
|
70
|
+
|
|
71
|
+
## Consequences
|
|
72
|
+
|
|
73
|
+
- The guard's threshold comes from the Harness container's limit, so it needs ADR-0060's limits first; layer 1 does not
|
|
74
|
+
and is fixed at once.
|
|
75
|
+
- The kill set includes a human's exec session open at that moment; the group kill would end it too. On the Instance
|
|
76
|
+
Harness that session is the only process the guard can kill, because a Menu-only Agent has no Working tools. No notice
|
|
77
|
+
follows: that pod holds no Workspace (ADR-0031, ADR-0062).
|
|
78
|
+
- A kernel kill (layer 5) while no Submission runs is told to the next Turn through the container's last end, which the
|
|
79
|
+
watch keeps until the container ends again. So two kernel kills before the next admission are told as one, and a
|
|
80
|
+
kernel kill that a later end of another kind replaces is not told. The guard fires at 90% of the limit, before the
|
|
81
|
+
kernel, so this window is the guard's misses alone.
|
|
82
|
+
- A memory fault is named in `jr2 status` and on the feed, not `Turn lost` (a restart for any other reason) or
|
|
83
|
+
"conversation lost" (a 404).
|