@wardby/cli 0.6.0 → 0.7.0

package/docs/coding-services.md

@@ -105,18 +105,18 @@ caller's principal id).
 
 An entry has:
 
-| Field             | Meaning                                                                                                                                                  |
-| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `name`, `version` | What a repository declares. Unique together.                                                                                                             |
-| `kind`            | `sidecar`: a fresh instance next to each run.                                                                                                            |
-| `image`           | The image, pinned by digest.                                                                                                                             |
-| `port`            | The port it listens on.                                                                                                                                  |
-| `serviceEnv`      | Environment for the service container. Per-run throwaway values only; never put a secret here.                                                           |
-| `testEnv`         | Variables the agent's shells receive. Upper-case names; names wardby uses itself (`PATH`, `HOME`, `PIP_*`, `WARDBY_*`, proxy settings, ...) are refused. |
-| `readiness`       | A command in the image that succeeds once the service accepts connections on `127.0.0.1`, with its period, timeout and failure threshold.                |
-| `resources`       | CPU (millicores), memory (MiB) and disk (MiB). Requests equal limits.                                                                                    |
-| `dataPath`        | The directory the service writes its data to; an empty volume in every run (memory-backed on Docker).                                                    |
-| `writablePaths`   | Other directories the image writes to (a socket directory, `/tmp`). The service's root filesystem is read-only.                                          |
+| Field             | Meaning                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
+| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `name`, `version` | What a repository declares. Unique together.                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
+| `kind`            | `sidecar`: a fresh instance next to each run.                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
+| `image`           | The image, pinned by digest.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
+| `port`            | The port it listens on.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
+| `serviceEnv`      | Environment for the service container. Per-run throwaway values only; never put a secret here.                                                                                                                                                                                                                                                                                                                                                                                                              |
+| `testEnv`         | Variables the agent's shells receive. Upper-case names. Names wardby uses itself are refused: `HOME`, `LANG`, `LC_ALL`, `PATH`, `SHELL`, `TMPDIR`, `USER`, `LD_PRELOAD`, `LD_LIBRARY_PATH`, `NODE_OPTIONS`, `NODE_PATH`, `PYTHONPATH`, `PYTHONHOME`, the proxy settings (`HTTP_PROXY`, `HTTPS_PROXY`, `NO_PROXY`, `ALL_PROXY`), `SSL_CERT_FILE`, `SSL_CERT_DIR`, and anything starting `WARDBY_`, `CODEX_`, `OPENAI_`, `ANTHROPIC_`, `PIP_`, `NPM_CONFIG_`, `UV_`, `GIT_`, `YARN_`, `PNPM_` or `COREPACK_`. |
+| `readiness`       | A command in the image that succeeds once the service accepts connections on `127.0.0.1`, with its period, timeout and failure threshold.                                                                                                                                                                                                                                                                                                                                                                   |
+| `resources`       | CPU (millicores), memory (MiB) and disk (MiB). Requests equal limits.                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+| `dataPath`        | The directory the service writes its data to; an empty volume in every run (memory-backed on Docker).                                                                                                                                                                                                                                                                                                                                                                                                       |
+| `writablePaths`   | Other directories the image writes to (a socket directory, `/tmp`). The service's root filesystem is read-only.                                                                                                                                                                                                                                                                                                                                                                                             |
 
 ### Built-in services
 
@@ -233,6 +233,8 @@ Protected-path entries also accept a leading `!` for your own exceptions:
 | A service never became ready                   | Launch   | failed, category `service_unready`         | "The `<name>` service didn't become ready, so the run couldn't start."                                                                                                                |
 | The run's changes touched a protected path     | Collect  | failed, category `protected_path`          | "its changes include `<path>`, which this agent may not edit, so none of its changes were kept. Ask again without changing that file, or have the repository owner make that change." |
 
+For a coding sub-agent of a sandboxed run, these refusals appear when the run starts instead of at dispatch, and the parent's delegation call receives the result.
+
 A refused run's `error` (from `get_run`) is the code followed by that sentence.
 The sentence reaches the requester on the run's status comment and, for a
 coding run started by another agent, in that agent's tool result. A
@@ -303,3 +305,16 @@ who maintain the catalog; see
 Existing agents are allowed no services until their owners add
 `codingProfile.services`, and every existing agent gets the `.wardby/`
 protected-path baseline without any change to its settings.
+
+### From 0.7.0: `YARN_*`, `PNPM_*` and `COREPACK_*` are reserved
+
+From Wardby 0.7.0 a service's `testEnv` can't set a variable starting
+`YARN_`, `PNPM_` or `COREPACK_`: the worker uses them to point yarn, pnpm and
+corepack at the package registry (see
+[pnpm and yarn](coding-packages.md#pnpm-and-yarn)). A custom catalog entry
+that sets one fails to load after the upgrade, and so does a run that
+declares it. Before you upgrade, list the catalog (`list_services`) and
+remove those variables from any entry's `testEnv` with `update_service`.
+After the upgrade such an entry can no longer be read or updated: delete it
+with `delete_service` and create it again without those variables. The
+built-in services set none.

package/docs/coding-worker-byo-images.md

@@ -285,6 +285,11 @@ These details explain the rules above.
   `/etc/profile` and the files in `/etc/profile.d/`, so that is where your own
   settings go. `/etc/profile` resets `PATH` to Debian's default, so add any
   other folder to `PATH` in your `/etc/profile.d/` file.
+  `/opt/wardby/bin` holds wardby's package-manager shim, linked as `npm`,
+  `pnpm` and `yarn`: it verifies the project's lockfile with the registry
+  before an install, then runs the image's own tool (or the release the
+  project pins, through corepack). Keep it first on `PATH`; see
+  [pnpm and yarn](coding-packages.md#pnpm-and-yarn).
 - **Network.** The tool runner reaches only wardby's package registry (npm and
   PyPI).
 - **Images on the Docker launcher are never pulled.** When wardby runs coding
@@ -322,6 +327,21 @@ current driver always disables. Rebuild any `workerImageRef` image on a
 current `driver-vN` digest (see "The driver image" above) before relying on
 either behavior.
 
+## The dependency cache needs a current driver
+
+The [dependency cache](coding-packages.md#dependency-cache) adds keys to the run
+input (`dependencyCache` and the workspace size). A worker built on an older
+driver base validates its input with a strict schema, so with the cache on
+for its agent every run fails at input (`worker_input_failed`) before the
+agent starts. For a Claude Code agent, a tool runner built on an older tool
+runner image doesn't fail the run, but it has no cache: its projects report
+`tool_unavailable`. Rebuild a `workerImageRef` image `FROM` the driver image
+(Codex) or tool runner image (Claude Code) of the release that includes the
+cache before you set `dependencyCache: true` on an agent that uses it.
+`create_agent` and `update_agent` return a warning when an agent has both.
+Images built `FROM` that release carry the cache module at
+`/opt/wardby/coding-worker/dependency-cache.mjs`; keep that path.
+
 ## What this doesn't cover
 
 wardby does not re-validate the contents of your built image beyond the

package/docs/coding-worker-isolation.md

@@ -74,12 +74,22 @@ again.
 If the stream still fails, the run's diagnostic (in the control-plane log,
 under the run's `coding_diag_…` id) names the cause as a fixed code, never the
 error text: `job_coding_stream_proxy_denied` (401/403 from the proxy),
-`job_coding_stream_rate_limited` (429), `job_coding_stream_upstream_error`
+`job_coding_stream_rate_limited` (429), `job_coding_stream_quota_exceeded`
+(the provider account is out of credit or over its spend limit; OpenAI answers
+that with a 429 too, and retrying never helps; see
+[the error article](../help/errors/coding-stream-quota-exceeded.md)),
+`job_coding_stream_upstream_error`
 (5xx), `job_coding_stream_timeout`, `job_coding_stream_proxy_unreachable`
 (the worker could not connect), `job_coding_stream_agent_exited` (the agent
 process exited without an HTTP error), or `job_coding_stream_failed` when none
 of these match. A refusal because the run's budget is used up ends the run as
 out of budget instead.
+Claude Code reports most API failures as a failed turn rather than a dropped
+stream; the Claude worker reads the error type its SDK reports and uses the
+same codes (a rate limit, an exhausted credit balance, an overloaded or
+failing provider, a refused credential, a timeout or an unreachable proxy). A
+failed turn with no such error, or with one none of these codes covers (for
+example an invalid request), is `job_coding_turn_failed`.
 
 The proxy side of the same request is in the coding proxy's log as
 `audit.*` events (`audit.request.reserved`, `audit.response.completed`,
@@ -240,21 +250,52 @@ expires on its own: each coding run dispatched before the expiry is traced for
 its whole life, and `get_run` shows `debugTrace: true` for it. Runs dispatched
 afterwards are not.
 
-A traced run's Codex worker writes every Codex stream event, and the full text
-of a stream failure (including the agent process's own error output and cause
-chain), to its **own log**, one JSON line each under a `debugTrace` key. On
+A traced run's worker writes its whole agent stream, and the full text of a
+stream failure (including the agent process's own error output and cause
+chain), to its **own log**, one JSON line each under a `debugTrace` key
+(`{ runId, at, kind, data }`, with `truncated: true` on a cut line). On
 Kubernetes that is the run pod's `worker` container log; on GKE, Cloud Logging
 keeps it after the pod is deleted. With the Docker launcher it is the worker
 container's log (`docker logs`), for as long as the container exists. Nothing from the trace goes to the database,
 GitHub, the run's error, or the control-plane log. Token-shaped values (the run
-capability, bearer tokens, API keys, GitHub tokens) are redacted, each line is
-capped at 16 KiB and each run at 2 MiB.
+capability, the package-registry token derived from it, bearer tokens, API
+keys, GitHub tokens) are redacted, and each line is capped at 16 KiB.
+
+- **Codex** runs trace every Codex stream event (`event`), turn and error
+  events (`turn_failed`, `error_event`), and stream failures (`stream_error`).
+- **Claude Code** runs (Wardby 0.7.0 or later) trace every Claude Agent SDK
+  message (`message`): the session's `system`/`init` message, each `assistant`
+  message, whose `tool_use` blocks carry the command given to `run_command`,
+  each `user` message, whose `tool_result` blocks carry that command's output
+  and error flag, and the final `result` message with its usage, cost and
+  duration. They also trace Claude Code's own error output (`sdk_stderr`) and
+  stream failures (`stream_error`). A command's duration is the difference
+  between the `at` of its `tool_use` message and that of its `tool_result`.
+- Both write `worker_error` when the worker itself fails.
+
+Each traced run's total is capped at 2 MiB by default. Once the cap is reached
+the worker writes one `limit_reached` line and stops tracing that run. An admin
+can raise or lower the cap for an agent, from 1 to 64 MiB, together with the
+trace or separately (Wardby 0.7.0 or later):
 
-The trace can contain prompts, model output and repository content. Turn it on
-only while diagnosing a failure, for as short a time as you can, and treat the
-pod logs of traced runs as sensitive. Tracing needs a worker driver image that
-supports it; a worker that predates it rejects a traced run's input as
-invalid.
+```
+update_agent { "id": "<agent id>", "codingProfile": { "debugTraceMinutes": 30, "debugTraceMaxMb": 20 } }
+```
+
+`debugTraceMaxMb` needs `agents:admin` too, and `null` restores the default.
+Changes are written to the same `coding.debug_trace.set` audit line. It is
+fixed on each traced run at dispatch, and `get_run` shows it as
+`debugTraceMaxMb` when it is not the default. A long trace can outgrow what
+your log pipeline keeps: Kubernetes rotates container logs (often at about
+10 MiB per file), so `kubectl logs` may show only the newest part of a large
+trace, while a log collector such as Cloud Logging on GKE keeps every line.
+
+The trace can contain prompts, model output, commands, their output and
+repository content. Turn it on only while diagnosing a failure, for as short a
+time as you can, and treat the pod logs of traced runs as sensitive. Tracing
+needs a worker image from a release that supports it (for Claude Code, 0.7.0
+or later). A Codex worker that predates it rejects a traced run's input as
+invalid; a Claude Code worker from 0.6.x runs the task but writes no trace.
 
 ## Container Policy
 
@@ -419,18 +460,20 @@ inert until an operator explicitly raises the ceiling above `CODING_DISK_MB`.
 A coding agent's profile carries an optional `workspaceDiskMb` (MiB; `null`
 means "use the deployment default `CODING_DISK_MB`"). It is snapshotted onto
 the `CodingRun` at dispatch time, so a later profile edit never changes an
-in-flight run's size, and it is capped by `CODING_MAX_DISK_MB`: a run whose
-snapshotted `workspaceDiskMb` exceeds the ceiling fails with
-`coding_workspace_disk_exceeds_limit`
-(`src/providers/executor/container.ts`, the `jobSpec()` check). This check
-runs after the workspace has already been cloned onto the control plane, so
-an over-ceiling agent still pays for a clone before failing, and the failure
-reaches the run record only as the sanitized `coding_failure_workspace:<id>`
-(the same generic bucketing every workspace/git-related failure gets) — not
-a distinctly labeled "refused" outcome, and not currently logged anywhere
-more diagnosable on the control plane. This is a known rough edge, not a
-security gap: no run ever exceeds the ceiling, it
-just fails less legibly than it could.
+in-flight run's size, and it is capped by `CODING_MAX_DISK_MB`.
+`create_agent` refuses a coding profile above the effective ceiling.
+`update_agent` checks the ceiling when setting a numeric `workspaceDiskMb`
+or changing an agent's kind to coding. Unrelated edits to an already-saved
+over-cap coding agent remain allowed. The error names the requested MiB, the cap, and
+`CODING_MAX_DISK_MB`, which the operator must raise to allow that size.
+A value exactly at the cap is allowed; `null` restores the deployment default.
+
+Agents already saved above the cap (including after an operator lowers it)
+still fail at run time with `coding_workspace_disk_exceeds_limit`.
+The run-time check runs after the workspace has been cloned onto the control
+plane and before the worker starts; the run record shows the sanitized
+`coding_failure_workspace:<id>`. Lower the profile's size or raise the
+operator ceiling before running the agent again.
 
 `CODING_MAX_CONCURRENT` (default `4`) caps coding runs that hold a slot at
 once, across every control-plane replica: the cap is enforced in Postgres
@@ -440,6 +483,12 @@ it starts, oldest first, when a slot frees (immediately in the process whose
 run finished, or on the scheduler leader's next tick). A newly dispatched run
 never takes a free slot ahead of an older queued run. A run still queued after
 `CODING_QUEUE_TIMEOUT_SEC` (default `3600`) fails with `coding_queue_timeout`.
+To end a queued run sooner, or a running one, call `stop_run` with its id: a
+queued run is recorded `cancelled` and never starts; a running run is recorded
+`cancelled` first, then its proxy session is cancelled and its job is stopped
+and removed. The cancelled state is checked again before anything is pushed,
+so a stopped run opens or updates no pull request, even when its worker had
+already finished (see [Stop or cancel a run](../help/stop-run.md)).
 Slot usage is derived from run state, so a crashed replica cannot leak slots:
 its runs are reconciled to `lost`, which frees them.
 
@@ -455,13 +504,33 @@ size (a service sidecar adds to a run's CPU and memory) and other pods share the
 quota. Set `KUBERNETES_RESOURCE_QUOTA` to the quota's name and the launcher
 checks, before a run claims a slot, whether the pod it would create fits the
 quota's free room. A run that doesn't fit is queued the same way as a run over
-`CODING_MAX_CONCURRENT`, and is retried when a run ends and on each leader tick;
-without the check, the API server refuses the pod and the run fails. The check
-reads one ResourceQuota by name (`get` on `resourcequotas` with that
+`CODING_MAX_CONCURRENT`, and is retried when a run ends and on each leader tick.
+The check reads one ResourceQuota by name (`get` on `resourcequotas` with that
 `resourceName` in the launcher's Role). If the quota is missing or can't be
 read, the launcher logs `kubernetes_resource_quota_missing` or
-`kubernetes_resource_quota_unreadable` once and launches as before. Size
-`CODING_MAX_CONCURRENT` to the quota too, so slots and quota agree.
+`kubernetes_resource_quota_unreadable` once and launches as before.
+
+The check can be stale: the quota may fill up between the check and the pod
+create, or `KUBERNETES_RESOURCE_QUOTA` may be unset. When the cluster still
+refuses the pod, the run does not fail. It returns to the coding queue (the
+control plane logs `coding run waiting for capacity: the cluster refused its
+pod`), keeps its place and its original queue clock, and is retried no sooner
+than about a minute later. It fails only when `CODING_QUEUE_TIMEOUT_SEC`
+expires, with `coding_queue_timeout`; a follow-up run on a pull request then
+also finishes the in-progress status it posted there, as failed. The queue stays first in, first out, so
+runs behind a refused run wait for it, and each retry happens about once a
+minute while the quota is full. A refused run shows `running` briefly, then
+`pending` again while it waits. When `KUBERNETES_RESOURCE_QUOTA` is unset and a
+pod is refused by a ResourceQuota, the launcher logs
+`kubernetes_resource_quota_unconfigured` once per process: set
+`KUBERNETES_RESOURCE_QUOTA` to that quota and grant the launcher `get` on it, so
+runs wait before they start instead of being refused at creation. The Docker
+launcher has no quota, so none of this applies there. In low-power mode, a run
+waiting in the coding queue keeps the coding proxy awake, as any queued run
+does.
+
+Size the quota for `CODING_MAX_CONCURRENT` times the run pod's requests, so
+slots and quota agree.
 
 Operating the queue across replicas:
 
@@ -521,7 +590,9 @@ the model, and a tag that breaks it is normalized to a lowercase slug (or
 dropped) instead of failing an otherwise finished run. GitHub finalization
 re-validates the tag independently. Linking an issue is not the tag's job: put
 a closing keyword such as `Resolves #37` in the task so the worker includes it
-in the summary, which becomes the pull request body.
+in the summary, which becomes the pull request body on every run,
+including follow-ups (see
+[How the pull request description is kept current](code-review-agents.md#how-the-pull-request-description-is-kept-current)).
 
 Some workspace folders are never collected from a run: `node_modules`, `.venv`,
 `venv`, `__pycache__`, `.pytest_cache`, `.ruff_cache`, `.mypy_cache`, `.tox`,
@@ -683,6 +754,10 @@ One pod per run, built by the canonical, deny-by-default policy in
   that points at the workspace's `.cache/tmp` on the disk-backed `storage`
   volume instead, so it scales with `workspaceDiskMb` rather than this tiny
   in-memory scratch.
+- A Claude Code run with `sessionResume` on also gives the worker's Claude
+  Code config directory (`/tmp/claude-config`, where the session file grows)
+  its own 80 MiB `medium: Memory` `emptyDir` (a tmpfs of the same size under
+  Docker), so a long session never fills `/tmp`. Only the worker mounts it.
 - `dnsPolicy: None` with `dnsConfig.nameservers: ["127.0.0.1"]` — **no DNS is
   configured for worker/agent pods at all.** The proxy is reached by name
   (`WARDBY_PROXY_URL=http://wardby-proxy:8787` — `CODING_PROXY_ALIAS` in
@@ -1105,7 +1180,7 @@ values. The list is kept only when every entry matches
 - **No end-to-end backpressure or host-side archive-size cap** once the
   exec WebSocket for a seed/collect transfer is connected — pre-existing,
   documented, not a regression of this milestone.
-- **An over-ceiling `workspaceDiskMb` fails late and generically** — see
+- **An already-saved over-ceiling `workspaceDiskMb` fails at run time** — see
   "Control Plane Configuration" above.
 - **Namespace handling must remain explicit in every overlay.**
   `deploy/kind-coding/manifests/base/kustomization.yaml`