@jr2/orchestrator 0.4.1 → 0.5.1

package/docs/reference/decisions/0039-image-garbage-collects-by-reachability.md

@@ -0,0 +1,115 @@
+# Image garbage collects by reachability
+
+[ADR-0038](0038-jr2-up-builds-every-image-it-deploys.md) made every tag a content address, which made garbage a
+by-product of normal work: ten Dockerfile iterations leave ten full images, and the prune it added collects only some of
+them — kind nodes only (the host daemon that built every image is never touched), at `jr2 down` only (the accumulation
+comes from iterating _while up_, which end-of-life pruning never sees), and scoped by tag **grammar**, which is
+ambiguous: `jr2-sandbox-<instance>-<name>` has no reserved delimiter, so instance `my` + image `extra-default` and
+instance `my-extra` + image `default` collide on one repo, and deleting `images/<x>/` orphans that image's tags because
+nothing derives their names any more. Each of those is a patch on the same wrong primitive: deciding ownership by
+parsing names.
+
+## Decision
+
+- **Ownership is a label, not a naming convention.** Every image `jr2 up` builds is stamped at build time —
+  `jr2.dev/kind` (`instance` | `sandbox` | `kit`) plus `jr2.dev/instance=<name>` on instance-owned ones — via
+  `docker build --label`, never in a Dockerfile (the user's file keeps zero jr2 knowledge, ADR-0037; the committed kit
+  Dockerfiles stay plain). Labels ride the image config through `kind load` into containerd, so both sides of the
+  transport can read provenance back. **The sweep touches labeled images and nothing else.** Name grammar stops being
+  load-bearing: the `my`/`my-extra` collision class and the orphaned-tags consequence in ADR-0038 both dissolve, because
+  deletion no longer needs to reconstruct a name — the image says who built it.
+- **Garbage is defined by reachability, not age or naming.** An image is needed iff a live root names it:
+  1. the `jr2-images` ConfigMap of **every** jr2 instance on the cluster (namespaces labeled `jr2.dev/instance`) — what
+     future Sandboxes will run;
+  2. every Sandbox CR's `spec.image` in those namespaces — a parked Workspace must survive a pod restart (`IfNotPresent`
+     cannot re-pull a local tag);
+  3. every pod's container images in those namespaces plus `jr2-system` — the orchestrator, Instance Harness, Custodian,
+     and operator actually running, mid-roll pods included, without naming Deployments one by one. The keep set is the
+     union, matched by whole ref after the one containerd normalization ADR-0038 already fixed (strip
+     `docker.io/library/`). A labeled image none of it names is garbage — user Sandbox Images and `jr2-*` kit images by
+     the same rule. **"Kit images are never pruned" dissolves into reachability**: a kit ref is kept because some
+     instance's map names it, and when the last instance leaves the cluster, kit images collect like everything else
+     instead of being permanent by fiat. A Pinned image — the Custodian's Envoy
+     ([ADR-0059](0059-a-harness-holds-stand-ins-and-the-custodian-holds-the-keys.md)) — is never taken at all: jr2 did
+     not build it, so it carries no stamp, and what jr2 did not stamp, jr2 does not sweep. It is a root besides (the
+     map's `custodian` key, every Sandbox CR's sidecars, every Harness pod), so no sweep would take it from under a live
+     pod even if it were stamped.
+- **`jr2 up` sweeps after a successful converge** — the moment the root set moves, which is where the iteration garbage
+  comes from. Nodes get a **one-generation grace**: refs named by the map this converge replaced stay one more round, so
+  the ConfigMap's kubelet propagation window cannot provision a just-swept ref. The **host daemon is swept
+  aggressively** (labeled ∧ not in the keep set ∪ this converge's refs): nothing runs from the host, its images are
+  scratch awaiting delivery, and BuildKit's cache — a separate store `docker rmi` does not touch — makes regenerating a
+  swept tag cost seconds. A failed converge sweeps nothing.
+- **`jr2 down` sweeps host and nodes after the namespace delete.** This instance's roots are gone, so its images are
+  unreachable by construction; another instance's roots still protect everything it shares, kit refs included. The
+  operator's image stays a root through its running pod unless `--all` takes the operator too. The exact-prefix matching
+  this replaces (`prunePlan`, the derived `jr2-sandbox-<name>-<x>:` list) is deleted, not kept as a fallback.
+- **Node policy is unchanged from ADR-0038's fix**: `crictl` cannot untag, so a node image id is removed (once) only
+  when **every** tag on it is unreachable; a mixed id is kept whole and reported. `docker rmi` on the host does untag,
+  so the host sweeps per tag with no id grouping. Both sides are delete-if-present, and a failed removal is reported and
+  skipped, never allowed to abandon the rest. The node **mechanics** are not unchanged: a `kind load`ed image is also
+  held under an `import-<date>@<digest>` ref CRI does not know, so `crictl rmi` alone exits 0 having reclaimed nothing —
+  removal reaches through to `ctr -n k8s.io images rm` for those names, and the report is confirmed by a re-list of the
+  node rather than by an exit code.
+- **The sweep narrates bytes, not counts** — `swept 4 image(s) (2.1 GB)` — because disk is the quantity the user feels,
+  and a silent GC plus one visible number is the entire intended interface. **`jr2 gc [--dry-run]`** is the same sweep
+  run off-cycle: an escape hatch for "disk is full now", not a step in any workflow; `--dry-run` prints the plan and
+  removes nothing. It never needs confirmation, because by construction it removes only what jr2 built and nothing
+  running or provisionable names.
+- **Registry-delivered copies are cache and sweep like everything else.** A pulled tag on a node (or the host's
+  just-pushed copy) is re-pullable from the registry, so unreachable copies go; the registry's own retention stays the
+  registry's business, as ADR-0038 drew the line.
+
+## Considered options
+
+- **Keep name-prefix pruning and extend it to the host** (the shape ADR-0038 grew fix by fix). Rejected: every fix
+  patched the same wrong primitive. Exact prefixes still collide (`my`+`extra-default` ≡ `my-extra`+`default`), deleting
+  an image folder still orphans tags, the kit exemption stays a fiat rule, and `up`-time iteration garbage stays
+  invisible. Reachability plus labels answers all four with one mechanism.
+- **A kind-local registry** so images live in one place and GC becomes registry retention. Rejected for now: real infra
+  (a registry container, containerd config on every node) bought mostly to avoid a sweep the CLI can do in a few
+  subprocess calls; the two-store shape (host builds, nodes hold copies) is a kind artifact this ADR manages rather than
+  rebuilds. Revisitable if a real registry becomes the default transport anyway.
+- **Age- or count-based GC** (`keep last N`, `older than 30d`). Rejected: wrong axis. Old is not unneeded (a parked
+  Workspace can hold a month-old ref alive) and new is not needed (a failed converge's tag is garbage at minutes old).
+  Reachability is the fact the others approximate.
+- **Sweeping at `down` only** (the ADR-0038 status quo). Rejected: the accumulation the prune exists for is produced by
+  iterating while up, so the collector must run where the garbage is made — after each converge.
+
+## Consequences
+
+- **Images built before this ADR carry no labels and are invisible to the sweep** — deliberately, since sweeping
+  unlabeled images means guessing by name again. The pre-existing strays (`jr2-workspace-*`, `:local` kit tags, old e2e
+  iterations) are cleaned once by hand or with the cluster; `jr2 up --force` re-tags current images with labels.
+- **The sweep reads cluster-wide** (namespaces labeled `jr2.dev/instance`, their ConfigMaps, Sandboxes, and pods), so
+  the CLI needs list access beyond its own namespace — true of the admin kubeconfig `jr2 up` already requires.
+- **A converge to another cluster can take a host image this converge built and has not used yet** — a second checkout's
+  converge, or this checkout's — because the host keep set only sees the current context's roots. `jr2 up` answers it
+  itself: a ref it counted on, its own build or a host-held skip
+  ([ADR-0041](0041-a-build-the-host-already-holds-is-not-spent-again.md)), that the daemon no longer holds at its use
+  (the `USER` read, the Effector image read, the `kind load` or the push) is built again from BuildKit's cache, in
+  seconds, and the use is retried, at most twice. A tag gone again after two rebuilds fails with docker's own error.
+- **CRI's image list can outlive containerd's refs**, because a removal that goes through `ctr` does not resync the CRI
+  image store: `crictl images` keeps answering for ids whose refs and content are gone. So the node read is filtered
+  against `ctr -n k8s.io images ls` — containerd's refs decide what is present, CRI's list only mirrors them — and
+  without that filter every id the sweep took would come back on the next plan, forever. The residual rows are cosmetic
+  (nothing can run from them, nothing is reclaimed by taking them again) and clear on a containerd restart; jr2 ignores
+  them by construction rather than reporting them.
+- **A converge in flight is not a root.** An image is needed iff a LIVE root names it — and a converge that has resolved
+  its refs but not yet applied them names them nowhere the cluster can see, so a sweep finishing in another namespace
+  can take a ref between its build and its delivery. On the host the rebuild above answers it; on the nodes nothing
+  does, and a ref taken after its `kind load` and before its map lands is an `ImagePullBackOff` at the next provision.
+  In the `@kind` tier under `--parallel` this is mostly out of reach: every scenario converges the same checkout, so
+  each worker's own `extraKeep` already holds the identical refs the others are building
+  ([ADR-0041](0041-a-build-the-host-already-holds-is-not-spent-again.md)'s build skip narrows the window further — a
+  warm converge builds nothing). It bites two instances on one cluster, or one instance converged twice across a source
+  edit. Recorded as a known limit; the fix — a claim the converge publishes BEFORE it builds, making an in-flight ref a
+  root like any other — is not taken here.
+- ADR-0038's `jr2 down` prune clause is **superseded by this ADR**; its `imagePullPolicy`/content-address reasoning
+  stands untouched, and ADR-0038's seal is what finally makes it true of the Instance's own artifact, its Assembly since
+  ADR-0069.
+- **Docker label inheritance makes derived images sweepable.** An image built `FROM` a jr2-labeled base inherits the
+  `jr2.dev/*` labels through the image config, so the sweep treats it as jr2's own — observed live with a brought ref
+  layered on a built Sandbox Image, collected by the next `jr2 gc`. A truly foreign brought image carries no such labels
+  and is untouchable as designed; one derived from a jr2 build should clear them (`LABEL jr2.dev/kind=`) or accept being
+  swept when no live root names it.

package/docs/reference/decisions/0041-a-build-the-host-already-holds-is-not-spent-again.md

@@ -0,0 +1,64 @@
+# A build the host already holds is not spent again
+
+[ADR-0038](0038-jr2-up-builds-every-image-it-deploys.md) keyed the build skip on the cluster's record — the
+`jr2.dev/images` annotation the previous converge stamped — deliberately: _"a claim about the CLUSTER'S RECORD, not the
+node's disk"_, because when tags were mutable, an image being present said nothing about it being current. A fresh
+namespace has no record, so it re-runs every `docker build` even when the host daemon holds every ref byte-for-byte —
+pure waste on a real cluster whenever a namespace is recreated, and ~7.4s per scenario across the `@kind` tier. The
+premise expired with ADR-0038's own seal: every tag is now an honest content address, so on the host that built it,
+**present implies current** is true rather than aspirational — asking the daemon is no longer trusting a guess, it is
+reading the fact the record only ever approximated.
+
+## Decision
+
+- **Two records, each answering the question it can vouch for.** The cluster's annotation answers for the _cluster_: a
+  ref it names was built and delivered by a converge that fully succeeded — skip the build **and** the delivery, exactly
+  as before. When the record is silent, the host daemon answers for the _host_: a labeled image whose tag equals the
+  resolved ref IS the build (the tag is a content address of the same inputs and platform set —
+  [ADR-0045](0045-the-platform-joins-the-image-address-and-the-cluster-chooses-it.md); an address silent about platform
+  let a daemon-default flip make one tag name two artifacts) — skip the build, **run the delivery**. A multi-arch ref
+  never lands in the daemon (buildx pushes straight to the registry), so the host skip cannot answer for it and that
+  build is spent. Only when both records are silent is docker spent. `--force` overrides both, unchanged in meaning:
+  rebuild and redeliver regardless of what anything claims.
+- **The disk answers the build question only, never the delivery one.** Delivery is already cheap-idempotent on both
+  transports — `kind load` skips a node holding the image id, and a push of layers the registry holds is a no-op per
+  layer — so the skip that would need _node_ or _registry_ observation buys seconds at the cost of a second stale-state
+  channel. The record keeps that job.
+- **The skip proves the build, not the floor.** The ADR-0037 floor is proven where the seat is known — the `preflight`
+  init step at provision, in the user's own image — so a converge, skipped or not, holds no preflight obligation and the
+  disk path re-proves nothing beyond tag-equals-content.
+- **One read, lazy, and advisory.** The daemon is asked once per converge at most (`hostImages()` — the same labeled
+  listing the sweep reads, ~13ms against the ~7.4s it saves), and only when some record check missed. A steady-state
+  converge still spends directory walks and no docker at all; ADR-0038's bullet stands. A listing that _fails_ reads as
+  "holds nothing": the check may only ever save a build, never fail a converge the build itself could survive — a truly
+  dead daemon fails at the build that follows, with docker's own error naming it.
+- **An unlabeled image never answers.** The listing is label-filtered, so a hand-built image that happens to wear the
+  right name is invisible and the build proceeds — the safe direction, and the same rule the sweep lives by
+  ([ADR-0039](0039-image-garbage-collects-by-reachability.md)): what jr2 did not stamp, jr2 does not trust or touch.
+
+## Considered options
+
+- **Replace the record with disk truth outright.** Rejected: the record answers a question disk cannot — what the
+  cluster was _delivered_ — and it is free (one kube read the converge already makes). Disk truth alone would re-run
+  every delivery every converge.
+- **Node-disk truth too** (`nodeImages()` before each `kind load`). Rejected: it duplicates the check `kind load`
+  already performs, at a `crictl` round trip per node, to skip a step measured fast.
+- **Registry-truth for the push path** (HEAD the manifest before pushing). Rejected for now: the push is idempotent, the
+  registry path has the record for steady state, and a registry API client is a new dependency for a case nothing
+  measured as slow.
+
+## Consequences
+
+- **The `@kind` tier's per-scenario converge stops paying the build tax.** A fresh namespace on a warm host skips all
+  five builds; the measured savings ride ADR-0010's numbers. It also narrows ADR-0039's sweeper-vs-builder window: a
+  converge that skips its builds holds nothing in flight for a sweep to take.
+- **The ADR-0039 hole is not widened.** The vulnerable window — a ref present on the host that no live root names,
+  between the skip decision and delivery — is the same window that already existed between a _build_ and its delivery,
+  and it fails the same way (the delivery fails loudly; re-run). In the `@kind` tier the delivered refs are identical
+  across scenarios and ride every sweeper's `extraKeep`. The two-instance case stays ADR-0039's recorded limit, and the
+  claim-published-before-building fix stays untaken.
+- **Recorded-but-absent is unchanged**: a record hit still skips everything, so a ref someone `docker rmi`'d _and_
+  removed from the nodes still surfaces as `ImagePullBackOff`, and `--force` is still the way back. The disk is
+  consulted only where the record was silent, never to audit it.
+- ADR-0038's skip bullet is **amended by this ADR**, not superseded: the record remains the first and cheapest answer;
+  what changes is that its silence stops being read as "the build never happened".

package/docs/reference/decisions/0042-ready-is-not-routable-an-admission-retries-its-connection.md

@@ -0,0 +1,142 @@
+# Ready is not routable: the two calls a turn starts with re-ask a Service that has not answered
+
+[ADR-0016](0016-agent-turn-mechanics-are-internal.md) sorted an Agent run's failures into classes and said which ones
+jr2 absorbs. Its **infra** class is stated as already-covered: _"provider-stream retry lives inside the turn in the
+Harness, and `wait` reconnects indefinitely from the offset ledger; a dead Harness surfaces as a fault"_. That sentence
+walks the turn from its stream backwards and never reaches the two steps in front of it. **Both legs that run BEFORE a
+model is ever asked were single, unretried `fetch`es** — the Orchestrator's admission POST to the Workspace Harness, and
+the Harness's `GET /agents/:iid/surface` that builds the turn's Menu — and either one rejecting settles the turn, which
+`actor.ts` reports as the terminal `agent.fault`. One dropped packet, one lost turn.
+
+The gap is not theoretical and not rare, because both of those calls dial a **Service that has only just started
+mattering**. The admission is the FIRST HTTP request jr2 ever sends over a Workspace's Service: the wrapper waits on the
+Sandbox CR's `phase: Ready`, which the operator computes from the POD's readiness, and the attach then reaches into the
+pod through the API server (`kubectl exec`) — neither proves the Service dialable. The surface read dials the
+Orchestrator's Service, which is between EndpointSlices every time the Orchestrator restarts, and restarting is ordinary
+([ADR-0007](0007-durable-machine-state.md)'s restore is built on it). Kubernetes programs the EndpointSlice behind a
+ClusterIP **after** the pod passes its probe, and until it does, kube-proxy REJECTs. **Ready is a statement about a pod;
+routable is a statement about a Service, and jr2 was reading the first as the second — at both ends of the same turn.**
+
+Measured: this is what cost the `@kind` tier its parallel default ([ADR-0010](0010-bdd-acceptance-tests.md)). Roughly
+one parallel run in three lost a scenario whose first turn never reached the scripted model, and the symptom was read
+for a long time as "a slow first turn". It is not. It is a lost turn, and the tier's failure dump caught both legs
+losing it:
+
+- **The admission**, three times. The pod's Harness answered `?view=history` with **404 — no conversation** (POST is
+  what creates one), the scripted model had received **zero** streaming requests, and the run settled
+  `{"outcome":"faulted","reason":"fetch failed"}`. In the same window the run-narrative echo — un-retried, and the only
+  other thing that dials that Service — logged the same `fetch failed` against the same address. Nothing had reached the
+  pod at all.
+- **The surface read**, once the admission was fixed and the tier ran again:
+  `submission settled failed: … {"error":"fetch failed"}` — the surface read's failure, in the scenario that RESTARTS
+  the Orchestrator. Which is the same defect one hop over: the Menu leg looks pod-local, and it is not — its hop leaves
+  the pod for the Orchestrator's Service.
+
+Parallelism causes neither; it multiplies the number of Services crossing into service at once, so it samples both
+windows more often. That is why the flake was degree-independent and serial runs never saw it.
+
+## Decision
+
+- **A never-delivered admission is re-sent.** `send` re-POSTs while the request demonstrably never left the host,
+  backing off on the same capped ladder `wait` uses. This is `wait`'s own rule applied one step earlier and for the same
+  reason: a request that could not connect is not a Harness that refused the prompt.
+- **Only never-delivered.** Admission is accept-and-queue
+  ([ADR-0027](0027-the-harness-is-jr2s-own-server-flue-retires-the-wire-stays.md)): a POST the Harness received but
+  could not answer has already queued a Submission, so re-sending it would run the turn **twice** — a worse failure than
+  losing it. The test is therefore the narrow structural question "did any byte reach the wire" (the errno's
+  `syscall`/`code`, read down the `cause` chain), never the broad "does this look transient". A reset connection is not
+  retried.
+- **One answer is re-sent too: a draining Harness's 503.** A Harness that received SIGTERM refuses an admission before
+  it reads or queues anything, and closes the connection
+  ([ADR-0031](0031-menu-only-agents-run-on-the-instance-harness.md)), so that 503 is as undelivered as a refused
+  connection — nothing ran, and the re-send dials the Service anew. It rides the same ladder and window. Every other
+  status is the Harness's word about this admission, and stays a fault.
+- **The window is bounded, where `wait`'s is not.** `wait` may reconnect forever because its Submission is already
+  admitted, so the lease owns the reporting ([ADR-0021](0021-workspace-continuity-is-a-lease-that-answers-back.md)'s
+  `workspace.lost`). Nothing is admitted yet at this seat, so there is no turn for a lease to be about: an address that
+  never answers is a fault this call must name itself, and it names the address it kept trying. **90 seconds — a
+  measurement, not a derivation.** The mechanism argues for far less: kube-proxy programs the EndpointSlice in well
+  under a second, and CoreDNS's 30s negative TTL is the only other obvious floor. Reasoning from the mechanism alone is
+  how this ADR first arrived at 60s, and 60s clears neither number the field has actually recorded —
+  [kubernetes#88986](https://github.com/kubernetes/kubernetes/issues/88986) measures **63s** on bare metal (a
+  SYN-retransmit ladder, 1-2-4-8-16-32, so those packets were DROPPED rather than refused), and
+  [kind#2280](https://github.com/kubernetes-sigs/kind/issues/2280) measures **up to 77s** with the EndpointSlices
+  already populated. The bound covers both. Its cost is paid only by an address that is genuinely wrong, and that one
+  faults with the address in hand either way.
+- **The ladder is jittered, because the callers arrive together.** Sandboxes that converge together cross the same
+  window together, and every Custodian in the cluster dials the same Orchestrator Service — so an un-jittered ladder
+  does not merely fail to help, it organizes the callers into a herd that retries and misses in lockstep. Each sleep is
+  drawn from the TOP HALF of its rung (equal jitter). The random half decorrelates them; the half that stays a floor is
+  the deliberate part, and it is what full jitter would give away — it is what still holds four clients off a Service
+  that is genuinely down.
+- **A transport failure says what it was.** `fetch` reports every one of them as the same three words, and
+  `reason: "fetch failed"` on a run is a diagnosis of nothing — the errno one level down (`connect ECONNREFUSED …`,
+  `getaddrinfo ENOTFOUND …`) is the whole answer. Both the admission fault and the echo's log line now carry it.
+- **The Harness re-asks an unanswered surface read, on ANY transport failure.** An unanswered read is the Harness's own
+  transport failure, or the Custodian's word that the Orchestrator never answered it (a 502 it marks as its own,
+  [ADR-0059](0059-a-harness-holds-stand-ins-and-the-custodian-holds-the-keys.md)), or no answer within 10 seconds. The
+  Custodian itself retries only a connection reset: any reset under the read, and under a pick or an ask only one before
+  the request was sent. It also drops a pooled connection after 4 seconds idle, before Node's 5-second keep-alive closes
+  it. A GET is idempotent, so the narrow never-delivered test the admission needs buys nothing here — there is no second
+  turn to accidentally start, so the rule is simply "an answered request is an answer". A 503 is the Orchestrator's word
+  that it is stopping, so it is re-asked too ([ADR-0026](0026-a-turn-that-is-over-is-never-prompted.md)). A 404 stays
+  ADR-0026's turn-is-over and a 403 stays a scope refusal: both are answers, and neither is re-asked.
+- **`deliver` does not retry, and that is not an oversight.** A failed pick reaches the model as a tool error it can act
+  on — pick again, or pick differently — so the turn survives one; and a POST that may have been delivered must never be
+  re-sent, because a duplicate pick is a duplicate transition. The asymmetry between the Menu's two calls is the same
+  one between `send` and `wait`: what may be re-asked is decided by what a second copy would do, never by how transient
+  the failure looks.
+- **The fault class is unchanged.** Exhausting either window is still ADR-0016's single terminal `agent.fault`, and
+  where it routes is still workflow policy. What changes is that it now means "this endpoint never answered", not "the
+  EndpointSlice was a few hundred milliseconds behind".
+
+## What this does not fix
+
+**An unrouted `agent.fault` is not this ADR's.** This ADR does not change what a workflow with no `agent.fault` route
+does with a lost Turn. [ADR-0074](0074-an-unrouted-mechanism-event-is-refused-at-load-and-faults-the-run.md) decides it:
+the walk refuses a state that invokes an Agent with no route on it or above it, and a fault that no transition takes at
+runtime ends the run `error`, with a `fault` that names the event, the state and the reason.
+
+## Consequences
+
+- The `@kind` tier's wired default becomes `--parallel 4` (ADR-0010 carries the measurements).
+- A genuinely wrong address at either seat now costs up to its window before it faults, instead of failing at once. That
+  is the intended trade: the address is right and the Service is late far more often than the address is wrong, and the
+  fault message names the address either way.
+- **The rule generalizes, and the next seat to need it should say so out loud.** Two hops needed this and were found one
+  at a time, each by a failure. The claim is not "retry the network"; it is that jr2 dials Services at the exact moment
+  they come into existence, and that a call which never got an answer has learned nothing worth acting on. Any new
+  jr2→Service call made at a lifecycle edge starts with that exposure until someone decides otherwise.
+- The retry is invisible to the workflow by construction (ADR-0016's absorption principle): no new event, no new budget
+  on the authoring surface, no bookkeeping in Machine context. **But absorbed is not the same as unmeasured, and the
+  difference is load-bearing.** Absorption alone would leave the tier benefiting from a window it cannot see: if
+  routability got twice as slow tomorrow, all 88 scenarios would still pass, a little slower, in silence, until the day
+  the window finally closed and the suite went red as a fresh mystery. Being unseeable is precisely how this class
+  survived three sessions, so each seat emits ONE line when a retry actually cost something —
+  `jr2.routability seat=… attempts=… ms=… last=… url=…` — and the `@kind` tier holds a budget against it, judged once
+  per worker in `AfterAll`. A log line, not a run-feed event: the authoring surface is untouched, so nothing above
+  contradicts itself. What the tier now asserts is not that the retry worked — the scenarios already assert that — but
+  **how much of the window is being spent**, which is what turns 90s from a number read off two GitHub issues into a
+  measurement of the cluster in front of us.
+- **The first live reading immediately corrected this ADR, which is the argument for having taken it.** The budget was
+  going to be attempts alone. The tier's first measurement was **2 attempts costing 10667ms** — one dropped SYN sitting
+  on undici's 10s connect timeout, not a ladder being climbed. So the refusal that dominates the cost here is a DROP,
+  not the kube-proxy REJECT this ADR leads with; both are real, and they cost in opposite currencies. A REJECT is
+  instant, so it spends attempts and almost no time. A DROP spends time and almost no attempts. An attempts-only budget
+  of 4 would therefore have waved through ~40s of a 90s window without a word. The very next run measured the other mode
+  — **3 attempts, 502ms, `ECONNREFUSED`** — which settled the shape: neither meter sees the other's failure, so the
+  budget is **30s or 8 attempts**, whichever trips first, and the line carries the final errno so nobody has to infer
+  the mode from the arithmetic. The two are calibrated to trip at comparable severity, which is what stops them being
+  one meter written twice: against the jittered 250ms→5s ladder, 8 attempts is ~9–18s of refusal, and 30s is three
+  connect timeouts. The first draft budgeted 4 attempts against an observed 3 — which is not a budget, it is a flake —
+  and only the second reading showed it.
+- **This is the ecosystem's answer, not a jr2 workaround**, which is worth recording because "retry" reads as a patch.
+  [kind#2280](https://github.com/kubernetes-sigs/kind/issues/2280) is this defect exactly — EndpointSlices populated,
+  connection still refused, up to 77s — and it establishes that waiting for the EndpointSlice is NOT sufficient, which
+  retires gating the CR's `phase: Ready` on it as a narrowing rather than a fix. Knative's activator arrives at the same
+  place from the other side: it
+  [probes the pod itself and treats its own successful probe as authoritative](https://knative.dev/blog/articles/demystifying-activator-on-path/)
+  regardless of what Kubernetes says, exactly so traffic can start before the control plane catches up. jr2's version is
+  that with one fewer moving part — the first real request IS the probe — which is only safe because `neverDelivered` is
+  what makes it re-sendable. A separate probe would add a round trip and put the race back (probe succeeds, the rule
+  changes, the real request lands).

package/docs/reference/decisions/0043-the-kit-is-tested-as-installed-a-local-registry-stands-in-for-npm.md

@@ -0,0 +1,170 @@
+# The kit is tested as installed: a local registry stands in for npm
+
+jr2's product shape ([ADR-0009](0009-cli-and-instance-interface.md)) is a globally installed CLI and a standalone
+Instance folder that just works — scaffold anywhere (`/tmp` included), install with whatever package manager the user
+already has, `jr2 up`. None of that path existed: the scaffold wrote `workspace:*` deps only the kit's own pnpm
+workspace can resolve, the bundle step was `pnpm deploy` (workspace members only), every package was `private: true`,
+and installed mode's kit image refs name published tags that exist nowhere. Every test tier ran checkout mode, so the
+mode users will actually run was the one mode nothing exercised. This ADR fixes the loop that closes that gap — and the
+loop is **permanent, not a pre-1.0 crutch**: "test 1.3.0 before it exists on npm" is the same problem forever.
+
+## The rule: fake the registry, never the mechanism
+
+The loop feeds the real machinery a local upstream — the same philosophy as
+[ADR-0038](0038-jr2-up-builds-every-image-it-deploys.md)'s scripted model provider (only the LLM is faked; here, only
+the registries are local). Concretely: an ephemeral [verdaccio](https://verdaccio.org) npm registry per loop run;
+`pnpm -r publish --registry` into it; `npm i -g @jr2/cli --registry` into a temp npm prefix; `jr2 init` in a temp dir
+outside any checkout and any git repo; the user's own package manager installs; `jr2 up` converges in installed mode.
+
+`npm link` was the tempting shortcut and is rejected for cause: a linked package is a symlink into the checkout, Node
+resolves modules by real path, so `detectKitCheckout()` finds the checkout and the CLI runs **checkout mode** — the
+branch under test never executes. Link also skips `npm pack` entirely (no `files:` check, no `workspace:*` rewrite),
+which is exactly the works-linked-broken-published failure class a release loop exists to catch. Tarball installs fail
+differently: npm cannot resolve inter-package deps (`@jr2/cli` → `@jr2/orchestrator`) to tarballs. A registry is the
+only stand-in that exercises the real chain.
+
+**An ephemeral registry deletes version bookkeeping.** Fresh verdaccio per run means `0.0.0` publishes cleanly every
+time — no `-dev.N` stamping, no manifest mutation before publish (a loop that edits manifests tests edited manifests).
+
+## What publishes, and the guard
+
+`@jr2/cli`, `@jr2/orchestrator`, `@jr2/agent-protocol` flip to `private: false` — the prod-resolution chain an instance
+pulls. `@jr2/harness` stays private: it reaches users as a **Kit image**, never via npm install
+([ADR-0027](0027-the-harness-is-jr2s-own-server-flue-retires-the-wire-stays.md),
+[ADR-0037](0037-an-instance-builds-its-sandbox-images-jr2-injects-the-harness.md)). This amends ADR-0009's "all
+`packages/*` publish to npm" — that line wrote down a mechanism, not the intent.
+
+Losing `private: true` loses the accidental-publish guard, so the guard moves into `publishConfig`:
+
+```json
+"publishConfig": { "registry": "http://localhost:4873" }
+```
+
+The manifest names the only registry it may publish to. `publishConfig` outranks CLI and env registry settings at
+publish time, so a stray `--registry` cannot leak past it, and a publish with no verdaccio up fails connection-refused.
+Publishing to npmjs becomes a deliberate, reviewable **edit** — changing that line is the "we are ready" commit — never
+an absence-of-flag accident. (**Amended by
+[ADR-0055](0055-a-release-is-a-pushed-tag-and-the-publish-guard-is-the-credential.md)**: the same precedence that made
+the line a guard made it unreleasable — the edit broke every reader of the guard at once. The line is gone; the guard is
+that no dev box holds an npmjs credential, and the release job publishes by trusted publishing on a pushed tag.)
+
+In the loop, Kit images need no registry at all on kind: build from the checkout, tag with the published names
+(`jr2-harness:<version>`), `kind load` — the same delivery checkout mode already uses. Registry-prefixed kit refs for
+non-kind clusters stay deferred (ADR-0038). (**Amended by
+[ADR-0044](0044-kit-images-live-at-a-canonical-home-a-self-host-mirrors-it.md)**: `kind load` bypassed the pull path, so
+the mode users run had no tier — the `@dist` bring-up now adds a local OCI registry, seeds it with `just kit-push`, and
+scenarios pull Kit images through `kitRegistry` like any self-host.)
+
+## The binary is the kit's one `.js` file
+
+Running the loop found what running checkout mode forever had hidden: Node's type stripping **refuses files under
+`node_modules`**, and an installed kit is nothing else — `jr2 --help` died on its own shebang. The image bundle already
+knew (its entrypoint runs through `tsx`, ADR-0038); the host-side binary had nowhere to learn it.
+
+So `bin/jr2.js` replaces `bin/jr2.ts`: it registers a `module.registerHooks` load hook that erases types with
+`ts-blank-space` — the same erasure ADR-0034 already makes for the Console — and then imports `../src/cli.ts`. What
+`ts-blank-space` cannot erase is what Node's own stripping rejects too, so this widens where the kit runs, never what it
+may be written in, and the repo stays zero-build. **One entry for both worlds**, for the same reason `jr2 init` grows no
+mode branch: a checkout-only `.ts` bin beside an installed-only `.js` bin would mean the binary users run is not the
+binary the tiers run.
+
+The loop's own state — registry storage, npmrc, throwaway global prefix — therefore lives **outside** the checkout. A
+prefix under the kit root is `npm link`'s rejected failure in another costume: `detectKitCheckout()` walks up from the
+CLI's real path, finds the kit above the prefix, and converges in **checkout mode** while reporting success. The publish
+script refuses a `JR2_DIST_DIR` inside the checkout, and the `@dist` converge asserts the mode line it expects.
+
+## The scaffold pins the kit that scaffolded it
+
+`jr2 init` renders `"@jr2/orchestrator"` and `"@jr2/cli"` at the **exact** running `KIT_VERSION` — exact, not caret,
+because 0.x minors break (the same reasoning that pinned pi exact, ADR-0027). One template serves both worlds; init
+grows no checkout/installed branch, because a branch means the tested output and the shipped output diverge.
+`templates/default` carries the same literal, so the byte-for-byte mirror test survives and becomes the version-bump
+tripwire. The checkout resolves the exact version to its own packages via `linkWorkspacePackages: true`; kit inter-deps
+stay `workspace:*` (publish rewrites them at pack). The scaffold names no package manager — no `packageManager` field, a
+PM-neutral install hint — and instances upgrade the kit by editing two dep lines, which pre-1.0 is a feature. (**Amended
+by [ADR-0056](0056-a-global-jr2-hands-off-to-the-instances-own-and-an-instance-has-one-kit-version.md)**: the two lines
+must name one number — a global `jr2` hands off to the Instance's own `@jr2/cli`, and a CLI whose orchestrator is not
+the Instance's refuses.)
+
+## The bundle installs from the lockfile, with the lockfile's own package manager
+
+The instance-image bundle ([ADR-0038](0038-jr2-up-builds-every-image-it-deploys.md)) is keyed off the **instance's own
+shape** — and not off the mode `jr2 up` detects for its Kit image refs. Asked in order: does the instance carry a
+lockfile of its own; if not, is there a `pnpm-workspace.yaml` at or above it. The instance's shape, not the CLI's
+provenance, is what says whether `pnpm deploy` can run at all, and the two keys disagree in both directions: a checkout
+CLI legitimately drives a standalone instance (a developer's `/tmp` folder), and an installed kit legitimately drives an
+instance nested in the user's own pnpm monorepo — which carries no lockfile of its own, because the workspace root holds
+it.
+
+_Amended:_ the key was first written as the workspace file alone — "is there a `pnpm-workspace.yaml` above me" — which
+is a fact about an ancestor, not the instance. A standalone instance committed inside a checkout whose
+`pnpm-workspace.yaml` does not list it went to `pnpm deploy`, which answers a `--filter` naming no member with "No
+projects matched the filters" and **exit 0**: no bundle, and the failure surfaced as an ENOENT from the seal. The
+lockfile is asked first because a member never has one and a standalone instance always does; and the deploy branch now
+checks the bundle exists, naming both ways out (join the workspace, or commit a lockfile) when it does not.
+
+- **Workspace member**: `pnpm deploy --legacy`, hoisted (`node-linker=hoisted`) like the standalone path, so the
+  Assembly of [ADR-0069](0069-the-orchestrator-is-a-kit-image-and-an-instance-publishes-its-assembly.md) is one flat
+  tree whichever path staged it. Its job — materializing workspace symlinks — only exists in a workspace. pnpm is a
+  **contributor prerequisite** (like go for the operator), and in the one installed-mode shape that reaches this branch
+  it is a prerequisite the user already met: they wrote the `pnpm-workspace.yaml` that selects it.
+- **Standalone**: stage a copy of the instance (minus `node_modules/`, `.jr2/`, `.git/`, and the credential files
+  `.env*`/`.npmrc` — `jr2 up` reads `.env` host-side into the Secrets it writes for the pods that spend each value,
+  ADR-0019 and ADR-0068, so a copy would bake a key into an image layer and into the tag addressing it) and run a
+  **frozen install from the lockfile**, dispatched on which lockfile is present: `package-lock.json` →
+  `npm ci --omit=dev`; `pnpm-lock.yaml` → `pnpm install --prod --frozen-lockfile` (with `node-linker=hoisted`, so the
+  bundle is flat real files regardless of PM); `bun.lock[b]` → `bun install --production --frozen-lockfile`. A private
+  registry reaches that install through the environment (`npm_config_registry`) or the user-level npmrc, never through
+  image content.
+
+The lockfile — not the user's `node_modules/` — is the input, and that is forced, not stylistic: the GitOps/CI path
+(ADR-0008's instance repo, the planned `jr2 build`) runs from a clean checkout where **no `node_modules` exists**, a
+copied tree bakes in accidents rather than declarations, and prod-pruning a copied tree means reimplementing resolution.
+Lockfile-as-input is also [ADR-0019](0019-one-converging-command-against-the-current-context.md)'s derivability rule
+applied to dependencies: the deployed bundle is a function of what `up` can see committed. The lockfile is therefore
+part of the instance contract; no lockfile is a hard error naming the supported three, two lockfiles is an ambiguity
+error.
+
+The package-manager "dependency matrix" collapses to nothing: lockfiles are proprietary formats, so supporting any PM
+mechanically means invoking the binary that speaks the lockfile present — and that binary's presence is guaranteed by
+the very thing that selects it (the user wrote the lockfile with it). jr2 itself depends on no package manager.
+
+**v1 ships npm, pnpm, and bun. Yarn is deliberately out**: one filename hides two incompatible generations (classic
+`--frozen-lockfile` vs berry `--immutable`), and berry defaults to PnP — no `node_modules` at all, which the image's
+resolution model cannot host. A `yarn.lock` gets a named rejection ("use npm, pnpm, or bun"), and adding it later is one
+dispatch row plus its tests, no design change.
+
+## The loop's two faces
+
+- **`@dist`**: a new opt-in e2e tag beside `@kind` (docker + kind + verdaccio), whose Rule is literally ADR-0009's
+  claim: a globally installed CLI converges a standalone instance. Publish happens once per suite run; scenarios isolate
+  by namespace; scenarios cover npm and pnpm instances (bun rides unit coverage on the dispatch until the tier needs
+  it), scaffolded in temp dirs with no git, converged, blown away.
+- **The manual loop**: a `just` recipe leaves verdaccio and the temp global install standing, so a developer plays the
+  way a user lives — `jr2 init /tmp/anything && npm install && jr2 up`, entirely outside the repo.
+
+## Consequences
+
+- ADR-0009's consequences: the instance-facing packages publish to npm; the Harness ships in a Kit image.
+- **No `@dist` scenario deploys a Custodian.** `ping` runs on no Harness pod, so the tier checks that the image map
+  re-homes the Custodian's Pinned image onto `kitRegistry`
+  ([ADR-0059](0059-a-harness-holds-stand-ins-and-the-custodian-holds-the-keys.md)) and pulls nothing. The fixture seeds
+  the registry with `just kit-push`, which builds Kit images only; a scenario that ran a Harness pod would need the
+  Envoy image mirrored there too (`jr2 kit push` does it).
+- Scaffolded instances pin exact and bump manually; a `jr2 upgrade` verb can exist later if that ever hurts.
+- The checkout's `pnpm-workspace.yaml` gains `linkWorkspacePackages: true`; `templates/default` trades `workspace:*` for
+  the exact version literal.
+- The npmjs `@jr2` scope must be claimed early regardless of readiness — the localhost guard protects against accidents,
+  not squatters.
+- **`jr2 up` still shells out to pnpm in one installed-mode shape**: an instance nested in the user's own pnpm
+  workspace. That is not the product dependency creeping back, because the thing that selects pnpm is a file the user
+  wrote with pnpm — the same self-selecting argument the lockfile dispatch already rests on.
+- **The published packages carry `files:` allowlists, and the over-include half is a unit assertion**
+  (`packages/cli/test/publish.test.ts`). A `files:` list fails loudly only when it OMITS something, so the `@dist` tier
+  is structurally incapable of catching a package that ships its test tree — and these are prod deps, unpacked into
+  every Assembly.
+- The staged bundle drops `node_modules/.pnpm-workspace-state.json` beside `.modules.yaml` (ADR-0038's seal): it is a
+  `lastValidatedTimestamp` and nothing else, so leaving it in re-addressed every pnpm bundle on every converge — a
+  pod-template change, and a snapshot restore for every live run, on a no-op `up`.
+- Everything the loop needs in-tree (public flags, localhost `publishConfig`, exact-pin scaffold, lockfile dispatch) is
+  the real shipping configuration. Nothing anywhere special-cases "testing" — which is the point.

package/docs/reference/decisions/0044-kit-images-live-at-a-canonical-home-a-self-host-mirrors-it.md

@@ -0,0 +1,85 @@
+# Kit images live at a canonical home; a self-host mirrors it
+
+[ADR-0038](0038-jr2-up-builds-every-image-it-deploys.md) gave installed mode published-`<kitversion>` Kit image refs and
+deferred the question of where those tags actually live "while nothing is published." The first installed-mode converge
+against a real cluster ends the deferral: `publishedKitRefs()` emits bare `jr2-harness:<kitversion>`, a node resolves
+that against `docker.io/library/`, and nothing is there — no registry a user pushes to can help, because the pod spec
+never names it.
+
+## Decision
+
+- **Kit images have a canonical public home: `ghcr.io/snapwich`.** `publishedKitRefs()` bakes it:
+  `ghcr.io/snapwich/jr2-harness:<kitversion>`, likewise the operator. Public GHCR egress is free to the project, so an
+  open-source kit can afford a home every user's nodes pull from — and only a canonical home makes
+  `npm i -g @jr2/cli && jr2 init && jr2 up` work with zero image plumbing, which is the product's promise. Pushing there
+  is part of the release train ([ADR-0019](0019-one-converging-command-against-the-current-context.md)): a release is
+  not done until the images for its npm version are at the home.
+- **A new config key `kitRegistry` re-homes the Kit refs** — `<kitRegistry>/jr2-harness:<kitversion>` — for self-hosted,
+  air-gapped, or mirror-only clusters. It re-homes the one Pinned image too, the Custodian's Envoy
+  ([ADR-0059](0059-a-harness-holds-stand-ins-and-the-custodian-holds-the-keys.md)):
+  `docker.io/envoyproxy/envoy:<tag>@<digest>` becomes `<kitRegistry>/envoy:<tag>@<digest>`. A mirror-only cluster needs
+  it as much as the Harness, and the digest does not move with the registry. It is deliberately **separate from
+  `registry`**: the two keys answer different questions — `registry` says where images _this converge builds_ go;
+  `kitRegistry` says where _published artifacts_ live. One key for both would force every private-`registry` user to
+  mirror Kit images they could have pulled from the home, making the common corporate case pay the rare air-gapped
+  case's cost. Like `registry`, it is deployment-varying → env.
+- **Getting Kit images into a self-hosted registry is deliberate and instance-less — never part of `jr2 up`.** Kit
+  images are shared by every instance on a cluster (and possibly by every instance in an org); updating them is a
+  deliberate act, not a side effect of converging one instance. Two arms, split by what each audience holds:
+  - **`jr2 kit push <registry>`** (in the binary, for the installed self-hoster): a **mirror**, nothing more —
+    registry-to-registry manifest copy of the Kit refs at the CLI's own `<kitversion>`, canonical home → target — and of
+    the Custodian's Pinned image, its upstream home → target, at the same digest — via
+    `docker buildx imagetools create`. That copies the full manifest list (every platform, exact digests) without
+    landing bytes on the host; a daemon-side `docker pull`/`push` would flatten a multi-arch image to the host's
+    platform and ship an amd64-only image to an arm64 cluster. A tag already present in the target is skipped and
+    reported: published version tags never move, so present implies current. The binary has **no build arm**: the npm
+    packages carry no Harness source, and an installed CLI that could build Kit images is exactly the patched-Harness
+    eject hatch [ADR-0027](0027-the-harness-is-jr2s-own-server-flue-retires-the-wire-stays.md)/ADR-0038 welded shut.
+  - **`just kit-push <registry>`** (in the checkout, for the kit developer and the release pipeline): build the Kit
+    images from source **multi-arch** (`docker buildx build --platform linux/amd64,linux/arm64 --push`) at the published
+    names into the given registry. This is how the canonical home gets its images at release — and how a dev-loop
+    registry gets images the home does not have yet. Single-arch pushes here would make the mirror faithfully copy the
+    deficiency to every self-host. It does not touch the Pinned image: it builds, and jr2 builds no Envoy.
+- **The `@dist` tier pulls Kit images through a real registry**
+  ([ADR-0043](0043-the-kit-is-tested-as-installed-a-local-registry-stands-in-for-npm.md)'s rule — fake the registry,
+  never the mechanism — applied to the second registry the product touches). The tier's kind bring-up adds a local OCI
+  registry (the upstream kind local-registry pattern: a `registry:2` container plus containerd `hosts.toml` on the
+  nodes); the fixture seeds it with `just kit-push`, scenarios set `kitRegistry`, and the nodes genuinely pull —
+  `kind load` at published names had bypassed the pull path entirely, leaving the mode users run covered by no tier,
+  which is the failure class ADR-0043 exists to delete.
+
+## Considered options
+
+- **No canonical home — self-host as the only model.** Rejected: every user's first `jr2 up` fails until they stand up a
+  registry and mirror the kit's images into it, inverting "the CLI does all the work." The bandwidth objection to a home
+  turned out to be moot: GHCR egress for public images is GitHub's cost, not the project's.
+- **One `registry` key prefixing everything.** Rejected above — the corporate-registry case pays the air-gapped case's
+  mirror cost for no benefit.
+- **Mirroring inside `jr2 up`** (absent tag → pull home, push self-host, converge on). Rejected: kit updates on a shared
+  cluster should be deliberate; a per-instance converge is the wrong actor to move an org-shared artifact, and the same
+  logic wants to run with no instance at all.
+- **One `jr2 kit push` with a checkout build arm** (checkout-is-the-signal, like `jr2 up`). Rejected: the build arm
+  needs the whole repo, so it belongs to the repo's own tooling — ADR-0038 already scoped `just` recipes as "shortcuts
+  for building images without a converge." The binary keeps only the arm its audience can use.
+- **Digest-pinned Kit refs** (`…@sha256:`), killing tag staleness outright. Rejected: installed mode would need a
+  converge-time registry resolve plus digest bookkeeping — a production mechanism purchased to serve a dev-loop
+  annoyance (see the accepted remainder).
+- **`imagePullPolicy: Always` on Kit containers.** Rejected: taxes every production pull to serve the same dev-loop
+  annoyance, inverting ADR-0038's "IfNotPresent is correct because tags are immutable."
+
+## Consequences
+
+- ADR-0038's deferred edge ("air-gapped clusters cannot pull published kit images") closes: the answer is the promised
+  registry prefix, spelled `kitRegistry`, fed by a deliberate mirror.
+- **Accepted remainder: a re-pushed dev tag is stale on any node that pulled the old one.** The loop pushes
+  `jr2-harness:0.0.0` repeatedly; `IfNotPresent` keeps the first bytes a node saw. Version tags are immutable everywhere
+  but the loop, and kit-source iteration is checkout mode's job — its content-addressed tags make staleness impossible
+  by construction. The installed loop tests distribution mechanics, not Harness code; when its content truly must move
+  on a persistent cluster, the escape is manual (`crictl rmi` on the node, or a throwaway cluster). `@dist` is immune —
+  its registry and cluster are ephemeral per run.
+- `just kit-push` needs a multi-platform buildx builder (qemu for the foreign arch). A checkout-mode converge against a
+  foreign-arch cluster needs no workaround: `jr2 up` derives the platform set from the cluster's nodes and passes
+  `--platform` explicitly ([ADR-0045](0045-the-platform-joins-the-image-address-and-the-cluster-chooses-it.md)), so
+  `DOCKER_DEFAULT_PLATFORM` steers nothing.
+- The home-cluster installed-mode loop becomes: `just dist-up` (npm side), `just kit-push zot.cluster.snapwich.net`
+  (image side), then an instance whose env sets `kitRegistry`.