@wardby/cli 0.5.3 → 0.7.0

package/docs/coding-worker-isolation.md

@@ -1,16 +1,24 @@
 # Coding Worker Isolation
 
-Wardby supports isolated Codex execution with Docker or Kubernetes and
-credential-separated Claude Code execution with Docker. Both launchers fail
-closed when the effective runtime does not match the reviewed policy.
+Wardby runs Codex and Claude Code coding workers through Docker or Kubernetes
+launchers. Both fail closed when the effective runtime does not match the
+reviewed policy. Claude Code separates its model-facing worker from its
+workspace-facing tool runner: Docker uses companion containers, while
+Kubernetes uses a native sidecar in the run pod.
 
 ## Security Boundary
 
-Coding-agent repositories and instructions are untrusted. The Docker daemon,
-host kernel, Wardby control plane, immutable worker image, and dedicated coding
-proxy are trusted. Containers are defense in depth rather than a VM boundary;
-production should run the Docker host on a dedicated worker node or VM with no
-production credentials beyond those required by the proxy.
+Coding-agent repositories and instructions are untrusted. The Wardby control
+plane, immutable worker images, dedicated coding proxy, and the platform
+enforcing container and network policies are trusted.
+
+### Docker launcher
+
+The Docker daemon and host kernel are trusted. Docker containers are defense
+in depth rather than a VM boundary. Production should run the Docker host on a
+dedicated worker node or VM with no production credentials beyond those
+required by the proxy. The Kubernetes launcher section below describes the
+corresponding pod and network rules.
 
 The Codex worker has one network attachment: a unique per-run internal bridge using
 Docker's isolated gateway mode. It has no default external route, published
@@ -66,12 +74,22 @@ again.
 If the stream still fails, the run's diagnostic (in the control-plane log,
 under the run's `coding_diag_…` id) names the cause as a fixed code, never the
 error text: `job_coding_stream_proxy_denied` (401/403 from the proxy),
-`job_coding_stream_rate_limited` (429), `job_coding_stream_upstream_error`
+`job_coding_stream_rate_limited` (429), `job_coding_stream_quota_exceeded`
+(the provider account is out of credit or over its spend limit; OpenAI answers
+that with a 429 too, and retrying never helps; see
+[the error article](../help/errors/coding-stream-quota-exceeded.md)),
+`job_coding_stream_upstream_error`
 (5xx), `job_coding_stream_timeout`, `job_coding_stream_proxy_unreachable`
 (the worker could not connect), `job_coding_stream_agent_exited` (the agent
 process exited without an HTTP error), or `job_coding_stream_failed` when none
 of these match. A refusal because the run's budget is used up ends the run as
 out of budget instead.
+Claude Code reports most API failures as a failed turn rather than a dropped
+stream; the Claude worker reads the error type its SDK reports and uses the
+same codes (a rate limit, an exhausted credit balance, an overloaded or
+failing provider, a refused credential, a timeout or an unreachable proxy). A
+failed turn with no such error, or with one none of these codes covers (for
+example an invalid request), is `job_coding_turn_failed`.
 
 The proxy side of the same request is in the coding proxy's log as
 `audit.*` events (`audit.request.reserved`, `audit.response.completed`,
@@ -232,21 +250,52 @@ expires on its own: each coding run dispatched before the expiry is traced for
 its whole life, and `get_run` shows `debugTrace: true` for it. Runs dispatched
 afterwards are not.
 
-A traced run's Codex worker writes every Codex stream event, and the full text
-of a stream failure (including the agent process's own error output and cause
-chain), to its **own log**, one JSON line each under a `debugTrace` key. On
+A traced run's worker writes its whole agent stream, and the full text of a
+stream failure (including the agent process's own error output and cause
+chain), to its **own log**, one JSON line each under a `debugTrace` key
+(`{ runId, at, kind, data }`, with `truncated: true` on a cut line). On
 Kubernetes that is the run pod's `worker` container log; on GKE, Cloud Logging
 keeps it after the pod is deleted. With the Docker launcher it is the worker
 container's log (`docker logs`), for as long as the container exists. Nothing from the trace goes to the database,
 GitHub, the run's error, or the control-plane log. Token-shaped values (the run
-capability, bearer tokens, API keys, GitHub tokens) are redacted, each line is
-capped at 16 KiB and each run at 2 MiB.
+capability, the package-registry token derived from it, bearer tokens, API
+keys, GitHub tokens) are redacted, and each line is capped at 16 KiB.
+
+- **Codex** runs trace every Codex stream event (`event`), turn and error
+  events (`turn_failed`, `error_event`), and stream failures (`stream_error`).
+- **Claude Code** runs (Wardby 0.7.0 or later) trace every Claude Agent SDK
+  message (`message`): the session's `system`/`init` message, each `assistant`
+  message, whose `tool_use` blocks carry the command given to `run_command`,
+  each `user` message, whose `tool_result` blocks carry that command's output
+  and error flag, and the final `result` message with its usage, cost and
+  duration. They also trace Claude Code's own error output (`sdk_stderr`) and
+  stream failures (`stream_error`). A command's duration is the difference
+  between the `at` of its `tool_use` message and that of its `tool_result`.
+- Both write `worker_error` when the worker itself fails.
+
+Each traced run's total is capped at 2 MiB by default. Once the cap is reached
+the worker writes one `limit_reached` line and stops tracing that run. An admin
+can raise or lower the cap for an agent, from 1 to 64 MiB, together with the
+trace or separately (Wardby 0.7.0 or later):
 
-The trace can contain prompts, model output and repository content. Turn it on
-only while diagnosing a failure, for as short a time as you can, and treat the
-pod logs of traced runs as sensitive. Tracing needs a worker driver image that
-supports it; a worker that predates it rejects a traced run's input as
-invalid.
+```
+update_agent { "id": "<agent id>", "codingProfile": { "debugTraceMinutes": 30, "debugTraceMaxMb": 20 } }
+```
+
+`debugTraceMaxMb` needs `agents:admin` too, and `null` restores the default.
+Changes are written to the same `coding.debug_trace.set` audit line. It is
+fixed on each traced run at dispatch, and `get_run` shows it as
+`debugTraceMaxMb` when it is not the default. A long trace can outgrow what
+your log pipeline keeps: Kubernetes rotates container logs (often at about
+10 MiB per file), so `kubectl logs` may show only the newest part of a large
+trace, while a log collector such as Cloud Logging on GKE keeps every line.
+
+The trace can contain prompts, model output, commands, their output and
+repository content. Turn it on only while diagnosing a failure, for as short a
+time as you can, and treat the pod logs of traced runs as sensitive. Tracing
+needs a worker image from a release that supports it (for Claude Code, 0.7.0
+or later). A Codex worker that predates it rejects a traced run's input as
+invalid; a Claude Code worker from 0.6.x runs the task but writes no trace.
 
 ## Container Policy
 
@@ -411,18 +460,20 @@ inert until an operator explicitly raises the ceiling above `CODING_DISK_MB`.
 A coding agent's profile carries an optional `workspaceDiskMb` (MiB; `null`
 means "use the deployment default `CODING_DISK_MB`"). It is snapshotted onto
 the `CodingRun` at dispatch time, so a later profile edit never changes an
-in-flight run's size, and it is capped by `CODING_MAX_DISK_MB`: a run whose
-snapshotted `workspaceDiskMb` exceeds the ceiling fails with
-`coding_workspace_disk_exceeds_limit`
-(`src/providers/executor/container.ts`, the `jobSpec()` check). This check
-runs after the workspace has already been cloned onto the control plane, so
-an over-ceiling agent still pays for a clone before failing, and the failure
-reaches the run record only as the sanitized `coding_failure_workspace:<id>`
-(the same generic bucketing every workspace/git-related failure gets) — not
-a distinctly labeled "refused" outcome, and not currently logged anywhere
-more diagnosable on the control plane. This is a known rough edge, not a
-security gap: no run ever exceeds the ceiling, it
-just fails less legibly than it could.
+in-flight run's size, and it is capped by `CODING_MAX_DISK_MB`.
+`create_agent` refuses a coding profile above the effective ceiling.
+`update_agent` checks the ceiling when setting a numeric `workspaceDiskMb`
+or changing an agent's kind to coding. Unrelated edits to an already-saved
+over-cap coding agent remain allowed. The error names the requested MiB, the cap, and
+`CODING_MAX_DISK_MB`, which the operator must raise to allow that size.
+A value exactly at the cap is allowed; `null` restores the deployment default.
+
+Agents already saved above the cap (including after an operator lowers it)
+still fail at run time with `coding_workspace_disk_exceeds_limit`.
+The run-time check runs after the workspace has been cloned onto the control
+plane and before the worker starts; the run record shows the sanitized
+`coding_failure_workspace:<id>`. Lower the profile's size or raise the
+operator ceiling before running the agent again.
 
 `CODING_MAX_CONCURRENT` (default `4`) caps coding runs that hold a slot at
 once, across every control-plane replica: the cap is enforced in Postgres
@@ -432,6 +483,12 @@ it starts, oldest first, when a slot frees (immediately in the process whose
 run finished, or on the scheduler leader's next tick). A newly dispatched run
 never takes a free slot ahead of an older queued run. A run still queued after
 `CODING_QUEUE_TIMEOUT_SEC` (default `3600`) fails with `coding_queue_timeout`.
+To end a queued run sooner, or a running one, call `stop_run` with its id: a
+queued run is recorded `cancelled` and never starts; a running run is recorded
+`cancelled` first, then its proxy session is cancelled and its job is stopped
+and removed. The cancelled state is checked again before anything is pushed,
+so a stopped run opens or updates no pull request, even when its worker had
+already finished (see [Stop or cancel a run](../help/stop-run.md)).
 Slot usage is derived from run state, so a crashed replica cannot leak slots:
 its runs are reconciled to `lost`, which frees them.
 
@@ -447,13 +504,33 @@ size (a service sidecar adds to a run's CPU and memory) and other pods share the
 quota. Set `KUBERNETES_RESOURCE_QUOTA` to the quota's name and the launcher
 checks, before a run claims a slot, whether the pod it would create fits the
 quota's free room. A run that doesn't fit is queued the same way as a run over
-`CODING_MAX_CONCURRENT`, and is retried when a run ends and on each leader tick;
-without the check, the API server refuses the pod and the run fails. The check
-reads one ResourceQuota by name (`get` on `resourcequotas` with that
+`CODING_MAX_CONCURRENT`, and is retried when a run ends and on each leader tick.
+The check reads one ResourceQuota by name (`get` on `resourcequotas` with that
 `resourceName` in the launcher's Role). If the quota is missing or can't be
 read, the launcher logs `kubernetes_resource_quota_missing` or
-`kubernetes_resource_quota_unreadable` once and launches as before. Size
-`CODING_MAX_CONCURRENT` to the quota too, so slots and quota agree.
+`kubernetes_resource_quota_unreadable` once and launches as before.
+
+The check can be stale: the quota may fill up between the check and the pod
+create, or `KUBERNETES_RESOURCE_QUOTA` may be unset. When the cluster still
+refuses the pod, the run does not fail. It returns to the coding queue (the
+control plane logs `coding run waiting for capacity: the cluster refused its
+pod`), keeps its place and its original queue clock, and is retried no sooner
+than about a minute later. It fails only when `CODING_QUEUE_TIMEOUT_SEC`
+expires, with `coding_queue_timeout`; a follow-up run on a pull request then
+also finishes the in-progress status it posted there, as failed. The queue stays first in, first out, so
+runs behind a refused run wait for it, and each retry happens about once a
+minute while the quota is full. A refused run shows `running` briefly, then
+`pending` again while it waits. When `KUBERNETES_RESOURCE_QUOTA` is unset and a
+pod is refused by a ResourceQuota, the launcher logs
+`kubernetes_resource_quota_unconfigured` once per process: set
+`KUBERNETES_RESOURCE_QUOTA` to that quota and grant the launcher `get` on it, so
+runs wait before they start instead of being refused at creation. The Docker
+launcher has no quota, so none of this applies there. In low-power mode, a run
+waiting in the coding queue keeps the coding proxy awake, as any queued run
+does.
+
+Size the quota for `CODING_MAX_CONCURRENT` times the run pod's requests, so
+slots and quota agree.
 
 Operating the queue across replicas:
 
@@ -513,7 +590,9 @@ the model, and a tag that breaks it is normalized to a lowercase slug (or
 dropped) instead of failing an otherwise finished run. GitHub finalization
 re-validates the tag independently. Linking an issue is not the tag's job: put
 a closing keyword such as `Resolves #37` in the task so the worker includes it
-in the summary, which becomes the pull request body.
+in the summary, which becomes the pull request body on every run,
+including follow-ups (see
+[How the pull request description is kept current](code-review-agents.md#how-the-pull-request-description-is-kept-current)).
 
 Some workspace folders are never collected from a run: `node_modules`, `.venv`,
 `venv`, `__pycache__`, `.pytest_cache`, `.ruff_cache`, `.mypy_cache`, `.tox`,
@@ -588,6 +667,7 @@ Kubernetes-specific settings (`src/config/providers.ts`,
 | `KUBERNETES_CONTEXT`            | (unset → in-cluster/default kubeconfig context) | Which kubeconfig context `ClientNodeKubernetesApi` connects with.                                                                                                                                                                                                                                     |
 | `KUBERNETES_RUNTIME_CLASS`      | (unset)                                         | e.g. `gvisor` on GKE. Unset means pods run without a sandboxing runtime class — logged once per launch as `kubernetes_runtime_class_unset` and development-only.                                                                                                                                      |
 | `KUBERNETES_RESOURCE_QUOTA`     | (unset)                                         | Name of the namespace ResourceQuota run pods count against, e.g. `wardby-coding` in the GKE overlay. When set, a run whose pod would not fit the quota's free room waits in the coding queue instead of failing at pod create. Needs `get` on that one `resourcequotas` object. Unset means no check. |
+| `KUBERNETES_PLATFORM`           | `generic`                                       | `generic`, `gke-autopilot` or `eks`. Selects the cluster's admission rules that run pods are conformed to. `gke-autopilot` and `eks` both require `KUBERNETES_RUNTIME_CLASS=gvisor`; see the platform profile section below.                                                                          |
 | `KUBERNETES_RUN_PRIORITY_CLASS` | (unset)                                         | PriorityClass for run pods and the preflight canary, e.g. `wardby-coding-run` in the GKE overlay. Must be an existing class and a DNS-1123 subdomain; `system-` classes are refused. Unset means no class (priority 0).                                                                               |
 
 The `CODING_CPUS` / `CODING_MEMORY_MB` / `CODING_PIDS` / `CODING_DISK_MB` /
@@ -674,6 +754,10 @@ One pod per run, built by the canonical, deny-by-default policy in
   that points at the workspace's `.cache/tmp` on the disk-backed `storage`
   volume instead, so it scales with `workspaceDiskMb` rather than this tiny
   in-memory scratch.
+- A Claude Code run with `sessionResume` on also gives the worker's Claude
+  Code config directory (`/tmp/claude-config`, where the session file grows)
+  its own 80 MiB `medium: Memory` `emptyDir` (a tmpfs of the same size under
+  Docker), so a long session never fills `/tmp`. Only the worker mounts it.
 - `dnsPolicy: None` with `dnsConfig.nameservers: ["127.0.0.1"]` — **no DNS is
   configured for worker/agent pods at all.** The proxy is reached by name
   (`WARDBY_PROXY_URL=http://wardby-proxy:8787` — `CODING_PROXY_ALIAS` in
@@ -798,6 +882,16 @@ landed on — cannot appear in it. That case reached a live Autopilot launch wit
 every dry-run-derived check passing, and is now pinned by tests. Treat a
 captured fixture as a lower bound on what a platform mutates.
 
+The `eks` profile is for Amazon EKS clusters with a dedicated gVisor node group.
+EKS has no managed sandbox runtime, so the cluster must provide a RuntimeClass
+named `gvisor` with handler `runsc` whose `scheduling` block selects nodes
+labelled `wardby.dev/sandbox=gvisor` and tolerates the taint
+`sandbox=gvisor:NoSchedule` on those nodes. Kubernetes' RuntimeClass admission
+merges that block into each pod; Wardby sets neither the node selector nor the
+toleration itself, and the `eks` profile forgives exactly that nodeSelector and
+that toleration and nothing else. Pod resources follow the generic rules, plus
+an explicit ephemeral-storage request and limit.
+
 ### The enforcement gate
 
 The Kubernetes API can create a NetworkPolicy object without that policy
@@ -911,9 +1005,9 @@ default 90,000ms):
 1. `platform` — pure configuration, checked before any cluster API call:
    `assertPlatformConfig` (`src/providers/jobs/kubernetes-platform.ts`) refuses
    a deployment that cannot work under `KUBERNETES_PLATFORM`. Under
-   `gke-autopilot` this means `KUBERNETES_RUNTIME_CLASS` must be `gvisor`
-   (gVisor is mandatory there — an unset or different runtime class is refused,
-   not warned about), and the effective `CODING_MAX_DISK_MB` must leave room
+   `gke-autopilot` and `eks`, `KUBERNETES_RUNTIME_CLASS` must be `gvisor` (an
+   unset or different runtime class is refused, not warned about). Under
+   `gke-autopilot` the effective `CODING_MAX_DISK_MB` must also leave room
    for the worker container's 1 GiB reservation inside Autopilot's 10 GiB pod
    ephemeral-storage ceiling. The same assertion runs again at process
    start-up in `buildConfiguredExecutor`
@@ -942,20 +1036,55 @@ default 90,000ms):
 5. `canary` — creates a real run pod + NetworkPolicy from the same builders
    as a live run, running a script that waits for policy enforcement (as
    above) then attempts DNS resolution, a connect to the proxy's deny port,
-   the internet (`1.1.1.1:443`), the metadata server, and the proxy itself —
-   requiring every one of the first four to fail and the proxy connect to
+   the internet (`1.1.1.1:443`), the cloud metadata endpoints (`169.254.169.254:80`,
+   the EKS Pod Identity agent `169.254.170.23:80` and its IPv6 address
+   `[fd00:ec2::23]:80`, and EC2's IPv6 metadata address `[fd00:ec2::254]:80`), and the proxy itself —
+   requiring every one of the first four kinds to fail and the proxy connect to
    succeed. Any other outcome, or a canary pod that itself fails to
    schedule/run, is `kubernetes_isolation_unsupported:canary`.
 
 This whole preflight is **memoized per launcher instance and its failure is
 sticky**: `KubernetesJobLauncher.runPreflight()` caches the first call's
-promise (`this.preflightResult ??= ...`, `kubernetes.ts:472-487`), including
+promise (`this.preflightResult ??= ...`, `kubernetes.ts:790`), including
 a rejection — so once a launcher process has seen preflight fail, every
 subsequent `launch()` in that process fails immediately with the same error
 without re-probing the cluster. A fresh preflight requires a new process
 (or, from the CLI, a fresh `wardby coding preflight` invocation, which is
 not memoized).
 
+**A long-running server process starts this same memoized preflight at
+start-up, not on the first coding run.** `wardby serve`, `wardby mcp` (both
+transports), and `wardby scheduler` each call `KubernetesJobLauncher.warmUp()`
+right after the executor is built, fire-and-forget: it runs the identical
+preflight `launch()` would otherwise run lazily, so the first coding run
+after a restart doesn't pay the preflight's own cost (a canary pod, routinely
+tens of seconds on a resource-constrained cluster such as `kind`).
+`wardby scheduler` needs this just as much as the other two — it dispatches
+scheduled coding runs through its own Kubernetes executor without ever
+starting an MCP server, so without this it would still pay the lazy cost on
+its first scheduled run. The result is logged once at start-up — an info
+line on success, a warning naming the failure code (e.g.
+`kubernetes_isolation_unsupported:<check>`) otherwise.
+
+**A start-up warm-up's failure is logged and retried on the next run, not
+left stuck until restart.** This is the one way a preflight failure is
+_not_ memoized: a transient cluster problem at process start (the API
+server briefly unreachable, a slow CNI not yet programmed, and the like)
+must not fail every coding run for the rest of that process's life, so a
+failed warm-up clears its own failed attempt once it has logged it, and the
+_next_ `launch()` runs the preflight fresh — succeeding if the cluster has
+since recovered. A `launch()` that was already waiting on that same
+in-flight warm-up attempt still fails with that attempt's error (it shares
+the same preflight call), exactly as it always has; it's only the attempt
+_after_ that one which retries. A preflight failure `launch()` triggers
+itself — because no warm-up ran, or because a warm-up's cleared failure was
+never retried before the next `launch()` found the cluster still broken —
+keeps the original behavior exactly: it stays memoized, failing every
+subsequent `launch()` in that process until it is restarted. This never
+runs for a one-shot CLI command (`wardby run`, `wardby coding preflight`,
+migrations, imports) — only for a process that stays up to serve or
+dispatch runs.
+
 **A hung pod create during preflight can leave a preflight pod and its
 NetworkPolicy behind.** If `createPod` never settles (rather than failing),
 the preflight's own timeout still fires and the caller sees `:timeout`, but
@@ -1051,7 +1180,7 @@ values. The list is kept only when every entry matches
 - **No end-to-end backpressure or host-side archive-size cap** once the
   exec WebSocket for a seed/collect transfer is connected — pre-existing,
   documented, not a regression of this milestone.
-- **An over-ceiling `workspaceDiskMb` fails late and generically** — see
+- **An already-saved over-ceiling `workspaceDiskMb` fails at run time** — see
   "Control Plane Configuration" above.
 - **Namespace handling must remain explicit in every overlay.**
   `deploy/kind-coding/manifests/base/kustomization.yaml`