@wardby/cli 0.5.3 → 0.7.0

package/help/dependency-cache.md

@@ -0,0 +1,143 @@
+---
+id: dependency-cache
+title: Dependency cache for coding runs
+summary: Restore npm node_modules from a verified snapshot before a coding agent starts, so runs skip the install.
+audience: operator
+tags: [coding-agents, packages, npm, node_modules, dependency-cache, cache, npm-ci, lockfile, snapshot, restore, blob-store, workspaces, tool_unavailable, budget_too_small, snapshot_skipped, timeout, CODING_CPUS, CODING_MEMORY_MB, disk, sha_mismatch, unsafe_archive, dependencyCache, worker_input_failed, workerImageRef]
+appliesTo: >=0.7.0
+---
+
+# Dependency cache for coding runs
+
+With the dependency cache on, a coding run's `node_modules` is prepared
+**before the agent starts** from a snapshot in a blob store, so the agent
+doesn't spend minutes on `npm ci`. It covers npm only. Off by default.
+
+## How it behaves
+
+For each eligible project the run reports one result in `get_run`
+(`dependencyCache.projects[]`):
+
+- `hit`: a snapshot for this exact lockfile was restored; nothing was
+  installed. A snapshot that fails to download or verify isn't used: the
+  project installs `cold` instead.
+- `nearest`: a snapshot for an older lockfile of the same repository and folder
+  was restored, then updated to the committed lockfile.
+- `cold`: no snapshot, so `npm ci` ran and the result was stored (when it fit the disk).
+- `skipped` or `refused`: nothing was prepared and the agent installs for
+  itself. A skip never fails the run; its `reason` says why.
+
+The agent's task says dependencies are pre-installed and not to run `npm ci`
+there (it would delete the prepared tree); `npm install <package>` is fine.
+A pull request gets a "Dependency cache: restored `node_modules` for ..." line
+for folders restored from a snapshot.
+
+## Why it is safe
+
+- Snapshots are built only by Wardby, before the agent starts. Then the upload
+  window is sealed: the worker seals it, and the coding proxy seals it itself
+  on the run's first model request, so the agent can't read or write the
+  cache.
+- A restore happens only after the coding proxy re-approves the committed
+  lockfile under the agent's current allowlist, release age and advisory
+  policy. A policy that now refuses a package means no restore.
+- Agents on the same repository share snapshots (the key is the repository,
+  project folder, lockfile and image, not the package policy), and each
+  restore is still gated by that agent's own current policy.
+- Keys come from the server, not the worker. Downloads are checked against the
+  SHA-256 and size the proxy recorded (that is the integrity check), the
+  archive is scanned, and the tree's recorded package versions are compared
+  with the lockfile before use. A damaged snapshot is deleted and rebuilt.
+  A restore skips re-scanning an archive whose scan result was recorded at
+  upload and whose digest matches; extraction still enforces every archive
+  rule.
+- The packages are the ones the agent's policy approves; the cache doesn't vet
+  their contents beyond what a normal install through the registry does.
+- Install scripts are not cached. Snapshots are built with `--ignore-scripts`;
+  `npm rebuild` runs after the seal, only when needed, with a minimal
+  environment.
+
+## When it applies
+
+- npm with `package-lock.json` `lockfileVersion` 2 or 3, at the repository root
+  and/or up to 4 projects up to 3 folders deep.
+- Ignored: `node_modules`, dot-folders, symlinks, folders that already have a
+  `node_modules`, and npm workspaces monorepos (root and members are skipped).
+- The agent needs a package allowlist.
+- Every worker image must come from a release that includes the cache: the
+  Codex worker images, the Claude Code worker image and its tool runner
+  images. An older Codex or Claude Code worker image fails the run at input
+  (`worker_input_failed`). Rebuild a `workerImageRef` image `FROM` that
+  release's driver image (Codex) or tool runner image (Claude Code) first;
+  `create_agent` and `update_agent` warn when an agent has both.
+
+## Turn it on
+
+1. On the coding proxy: `WARDBY_DEPENDENCY_CACHE=true` plus a blob store
+   (`BLOB_STORE=gcs|s3|local` and its bucket or directory variables). GKE:
+   `dependency_cache_enabled` and `dependency_cache_bucket_name` in Terraform,
+   then `WARDBY_DEPENDENCY_CACHE=true deploy/gke/up.sh`. AWS: the same Terraform
+   variables, plus the `BLOB_STORE_S3_*` proxy variables. kind:
+   `WARDBY_DEPENDENCY_CACHE=true bash deploy/kind-coding/up.sh` (not durable).
+   Docker quickstart: add `WARDBY_DEPENDENCY_CACHE=true` to `.wardby/.env`
+   (requires wardby 0.7.0 or later).
+2. GKE: nothing extra. `up.sh` grants the proxy the `DependencySnapshot`
+   table before the rollout.
+3. Per agent, once the worker images are current: `update_agent` with
+   `codingProfile.dependencyCache: true`. This needs `packages:approve` or
+   `agents:admin`.
+
+## Disk use
+
+The cache works within a budget: the run's workspace disk
+(`workspaceDiskMb`) less what the checkout uses and a margin. Restores and the
+snapshot archive are held to it while they are written. An install is checked
+only before it starts: at least 1 GiB must be left (or twice the project's
+tree when a snapshot of it was seen), otherwise the project is
+`budget_too_small` and the agent installs for itself. npm's writes during the
+install itself aren't bounded. After its own install the worker deletes npm's
+download cache (`_cacache`), then archives the tree for upload only if the
+archive fits what is left. A snapshot skipped for space keeps the installed
+tree and the `cold` or `nearest` result; the worker log says
+`snapshot_skipped` (reason `budget`), not `get_run`.
+
+Tuning, on the proxy: `WARDBY_DEPENDENCY_CACHE_MAX_SNAPSHOT_MB` (1024),
+`_REPO_CAP_MB` (4096), `_TOTAL_CAP_MB` (20480), `_MAX_AGE_DAYS` (14, counted
+from creation, so snapshots are rebuilt periodically) and `_UPLOAD_SECONDS`
+(900). On GKE and AWS keep the Terraform `dependency_cache_max_age_days` equal
+to `_MAX_AGE_DAYS`; the bucket also deletes objects older than that plus 7 days.
+
+## Troubleshooting
+
+Read `dependencyCache` in `get_run`; then match the `reason`. A rejected `hit`
+or `nearest` snapshot is not a reason: the project shows `cold`.
+
+| Reason                                                                                       | What to do                                                                                                                                                                                                                                         |
+| -------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `no_lockfile`, `lockfile_unreadable`, `no_package_json`, `no_dependencies`                   | Nothing to cache in that folder.                                                                                                                                                                                                                   |
+| `lockfile_version`                                                                           | Regenerate the lockfile with npm 7 or later.                                                                                                                                                                                                       |
+| `workspaces`, `workspace_member`, `shrinkwrap_present`                                       | Not supported; the agent installs itself.                                                                                                                                                                                                          |
+| `node_modules_present`, `cache_dir_unsafe`                                                   | Remove the committed `node_modules` or `.cache` (symlink) from the repository.                                                                                                                                                                     |
+| `too_many_projects`                                                                          | Only 4 projects per run are prepared.                                                                                                                                                                                                              |
+| `no_registry`, `plan_failed`, `plan_refused`                                                 | Give the agent a package allowlist and fix refused packages; see [Approve packages for coding agents](help://coding-packages). A plan the proxy left incomplete is retried for up to 10 minutes (at most 5 attempts per project, 8 plans per run). |
+| `cache_unavailable`, `plan_required`, `sealed`, `lookup_failed`, `download_failed`           | Check `WARDBY_DEPENDENCY_CACHE` and `BLOB_STORE` on the proxy, the store's permissions and the proxy log.                                                                                                                                          |
+| `size_mismatch`, `sha_mismatch`, `unsafe_archive`, `inconsistent`                            | The snapshot was not trusted; the project installed `cold` instead. It is rebuilt when it ages out or is deleted. Repeats point at the blob store.                                                                                                 |
+| `too_large`, `over_budget`, `budget_unknown`, `budget_too_small`                             | Raise `WARDBY_DEPENDENCY_CACHE_MAX_SNAPSHOT_MB` or the agent's `workspaceDiskMb` (see [Disk use](#disk-use)).                                                                                                                                      |
+| `install_failed`, `timeout`, `cancelled`, `internal_error`, `worker_error`, `report_invalid` | The agent installs for itself; check the worker log. The preparation gets at most 20 minutes or half the time left before the run's deadline. On Docker a huge restore can exhaust the tmpfs inode limit before its bytes.                         |
+| `tool_unavailable`                                                                           | Claude Code tool-runner image predates the cache or is custom; rebuild it from the current tool-runner image. The proxy still seals on the run's first model request.                                                                              |
+| `already_started`, `invalid_deadline`                                                        | Not expected: a command ran before preparation, or the worker and tool runner images are from different releases.                                                                                                                                  |
+| `worker_input_failed` (the run fails)                                                        | A Codex or Claude Code worker image, or a `workerImageRef` image, older than the cache. Update the images, or rebuild the custom image `FROM` the current release, before turning the cache on.                                                    |
+
+Every run of a large project installing `cold` again usually means its
+snapshot is never stored. Check the worker log for `snapshot_skipped` with
+reason `timeout`: its `limit` says which bound ran out. For `pack` or
+`deadline`, give coding pods more CPU (`CODING_CPUS=2`); for `upload_window`,
+raise `WARDBY_DEPENDENCY_CACHE_UPLOAD_SECONDS`. A large tree (about 1 GiB) can
+also need more than the default 2048 MiB of memory to install under gVisor;
+raise `CODING_MEMORY_MB`.
+
+`rebuild` of `failed`, `timeout` or `unsealed` means native add-ons may be
+unbuilt; the agent can run `npm rebuild`.
+
+The full reference, with every limit, is
+[`docs/coding-packages.md`](../docs/coding-packages.md#dependency-cache).

package/help/deploy-gke.md

@@ -3,7 +3,7 @@ id: deploy-gke
 title: Deploy Wardby on GKE Autopilot
 summary: Use the supported Google Cloud path for a private database, isolated coding workers, and HTTPS ingress.
 audience: operator
-tags: [deployment, gke, gcp, kubernetes, production, secrets, jira]
+tags: [deployment, gke, gcp, kubernetes, production, secrets, jira, upgrade, database-grants]
 appliesTo: >=0.2.1
 ---
 
@@ -25,23 +25,37 @@ The deployment process is:
 1. Install `gcloud`, Terraform, Docker with `linux/amd64` support, `kubectl`,
    Helm, Node.js 24, and authenticate to the target project.
 2. Configure `deploy/gke/terraform.tfvars`, apply Terraform, and prepare the
-   Gateway's address, certificate map, Cloud Armor policy, and DNS record.
+   Gateway's address, certificate map, Cloud Armor policy, and DNS record. On
+   a brand-new project, run `deploy/gke/bootstrap-database-iam.sh` once after
+   `terraform apply`, so the migrator has its database grants before the
+   first migration.
 3. Put first-time values in an untracked `.env.local`; `deploy/gke/up.sh`
    seeds Secret Manager without overwriting existing production values.
    Jira settings are optional there, all or none; see [Jira](jira.md).
-4. Run `HOSTNAME=wardby.example.com deploy/gke/up.sh`, then verify DNS,
-   certificate issuance, database IAM bootstrap, and service health.
+4. Run `WARDBY_HOSTNAME=wardby.example.com deploy/gke/up.sh`, then verify DNS,
+   certificate issuance, and service health.
+   Use `WARDBY_HOSTNAME`, not `HOSTNAME` (shells set that to the machine name);
+   the script refuses local names such as `*.lan` or `*.local`, and asks you
+   to confirm (or set `WARDBY_HOSTNAME_CHANGE=<new hostname>`) before moving
+   a deployment to a different hostname.
 
-When a release changes `deploy/gke/database-grants.sql`, re-run the database
-grants bootstrap **before** deploying that release, so the proxy role can
-already write the tables and columns it adds (such as per-model usage for
-[cost attribution](cost-attribution.md), or a run's live turn count). Until the
-grants are applied, the coding proxy's writes are refused and coding runs fail.
+To upgrade, re-run `deploy/gke/up.sh` from the new release. It applies the
+migrations, then the database grants (`deploy/gke/database-grants.sql`), and
+only then rolls out the new pods, so a release that adds tables or columns the
+coding proxy uses (such as per-model usage for
+[cost attribution](cost-attribution.md)) needs no manual grants step. If the
+grants fail, `up.sh` stops before the rollout with a non-zero exit;
+`deploy/gke/bootstrap-database-iam.sh --check` shows the refused statement
+without changing anything. See
+[Upgrading an existing deployment](../docs/getting-started-gke.md#upgrading-an-existing-deployment).
 
 The Kubernetes launcher runs Codex and Claude Code coding workers; a Claude
 Code pod adds a tool-runner sidecar with its own image. Configure an identity
 provider and GitHub App before allowing people to use the public endpoint.
 
+To lower the idle cost of a lightly used deployment, see
+[Low-power mode](../docs/getting-started-gke.md#low-power-mode) (Wardby 0.6.0 and later).
+
 ## Plan it with your assistant
 
 If you are an assistant connected to Wardby over MCP and the user asked to plan

package/help/deployment-targets.md

@@ -1,7 +1,7 @@
 ---
 id: deployment-targets
 title: Choose a deployment target
-summary: Pick the supported Wardby deployment path and understand its operational boundary.
+summary: Compare local, self-hosted, and GKE deployment paths and their operational boundaries.
 audience: operator
 tags: [deployment, docker, gke, aws, production]
 appliesTo: >=0.2.1
@@ -9,31 +9,42 @@ appliesTo: >=0.2.1
 
 # Choose a deployment target
 
-Wardby has one local path and two production-ready deployment shapes:
+Wardby has a local development path, a self-hosted container deployment
+reference, and a supported GKE reference deployment. The references provide
+different parts of a production environment; operators must complete the
+remaining controls and verification before accepting production work.
 
 - **Local development:** `wardby quickstart` runs the control plane locally
   with its portable PostgreSQL container. It is the best place to evaluate,
   develop agents, and connect a local Codex or Claude Code client.
-- **Production container baseline:** run the published production image with
-  PostgreSQL, HTTPS ingress, durable storage, backups, and an operator-owned
-  identity provider. The Compose and Caddy configuration is a reference
-  baseline, not a managed platform.
+  `deploy/kind-coding` is a local Kubernetes proof harness, not a production
+  deployment.
+- **Self-hosted container deployment** (e.g., Docker Compose on a VM or
+  on-premises host): Wardby supplies runtime and migration images plus a
+  Compose/Caddy edge reference. The operator supplies PostgreSQL, secrets,
+  network and egress controls, backups and restore testing, monitoring, and a
+  separate coding-controller host. Configure authentication for the deployment.
 - **Google Kubernetes Engine Autopilot:** the supported Google Cloud path. It
-  provisions GKE, private-IP Cloud SQL, Artifact Registry, isolated Codex and
-  Claude Code workers, HTTPS Gateway, and GCP-native secret and network
-  controls. See [Deploy on GKE](deploy-gke.md).
+  provisions GKE, private-IP Cloud SQL with backups and point-in-time recovery,
+  Artifact Registry, isolated Codex and Claude Code workers, HTTPS Gateway, and
+  GCP-native secret and network controls. Operators still need to choose
+  availability, test recovery, and configure alerts and retention. See
+  [Deploy on GKE](deploy-gke.md).
 
-AWS is supported as a portable runtime target and has a Bedrock Claude adapter,
-but Wardby does not ship a native AWS deployment module. Other cloud providers
+AWS can run the portable runtime and has a Bedrock Claude adapter, but a native
+AWS deployment module is planned and has not shipped. Other cloud providers
 can run the production container image with equivalent database, ingress,
 identity, secret, isolation, and observability controls; that infrastructure is
-operator-owned.
+operator-owned. On Amazon EKS, set `KUBERNETES_PLATFORM=eks` (gVisor node group
+required) and, for RDS, `DATABASE_AUTH=rds-iam`; see "Database
+authentication" in
+[`docs/security-deployment.md`](../docs/security-deployment.md).
 
 The older `deploy/gcp` Cloud Run module is deprecated. Do not choose it for a
 new installation.
 
-Before going live, complete the deployment security checklist and make a
-backup, upgrades, alerting, and incident-response plan. Read
+Before going live, complete the release verification and security deployment
+checks, and make plans for backups, upgrades, alerts, and incident response. Read
 [`docs/getting-started.md`](../docs/getting-started.md) for the local and
 container setup, and [`docs/security-deployment.md`](../docs/security-deployment.md)
 for the production controls.

package/help/errors/budget-exhausted.md

@@ -0,0 +1,187 @@
+---
+id: errors/budget-exhausted
+title: Run stopped out of budget (budget_exhausted)
+summary: A coding run stopped because its next model request did not fit what was left of its budget; get_run's budgetEvents says which limit and why.
+audience: operator
+tags: [budget, budget_exhausted, coding_budget_exhausted, overrun, allowedOverrunPercent, overrun_unavailable, draft]
+appliesTo: ">=0.7.0"
+---
+
+# Run stopped out of budget
+
+## What happened
+
+Before each model request the coding proxy holds an amount against the run's
+budget. The first request of a conversation (and any fallback) holds the worst
+case: the whole request body as new input plus the largest reply the request
+allows, priced at the model's output rate. After a mostly cached reply in the
+same conversation, the cached part of the prompt is held at the cache-read
+rate instead, so the hold is much smaller. A request whose hold does not fit in what is
+left is refused, and the run ends `budget_exhausted` (its error reads
+`coding_budget_exhausted: <sentence>`). The run stopped before a request whose
+hold did not fit. A cache miss can make a settled reply cost more than its
+hold, up to its worst case (see [Holds vs actual spend](#holds-vs-actual-spend)).
+
+## Find out why
+
+Call `get_run` and read `budgetEvents`. Each event has:
+
+- `kind`: `refused`, or `overrun_unavailable` for an overrun top-up its group
+  or run tree had no room for (see [Overrun not available](#overrun-not-available)).
+- `limit`: which cap bound the run.
+  - `run`: the agent's own `budgetUsd`.
+  - `run-tree`: what the lead agent had left for its whole run tree.
+  - `day`, `week`, `month`: the budget group's cap for that period.
+- `at`: when it was recorded.
+- `details`: for a refused model request, the budget, the amount spent, the
+  holds of requests in flight, the refused hold, the model and the turn. For a
+  refusal at dispatch, the cap, the amount spent, the amount held and
+  `heldRuns`, the number of runs holding budget. A refused model request also
+  carries `refusedHoldBasis` (`cached` or `full`) and `refusedHoldCeilingUsd`
+  (the worst-case hold for that request). Both also carry
+  `groupName` (the agent's budget group, when it has one) and, for a
+  `run-tree` limit, `leadAgentName`. A refused model request also carries
+  `agentBudgetUsd`, the agent's own `budgetUsd`. An `overrun_unavailable`
+  event carries the full cap or tree's `capUsd`, `spentUsd`, `heldUsd` and
+  `leftUsd`, the run's `baseBudgetUsd`, `grantedUsd` and `limitUsd`, and
+  `groupName`, the group whose cap was full (which can be the lead's).
+
+The run's error, and a lead's sub-agent result (in its `budget` field), carry
+the same facts as one sentence. Pull request and issue comments and check
+summaries show the amounts and the budget group's name only.
+
+## Holds vs actual spend
+
+A hold is not a charge. The run is charged the reply's real cost, and the hold
+is released when the reply is settled. A hold is larger than a typical reply
+because it must cover the whole output allowance (`maxOutputTokens`), and
+because the first request of a conversation, and any fallback, assumes the whole
+body is new input. Fallbacks apply when there is no previous reply in the
+conversation, the model changed, the previous reply was interrupted or mostly
+uncached, the previous request in the conversation was sent more than 5 minutes
+earlier, or the body got smaller.
+
+If the cache misses, a reply can cost more than its hold. It is still settled
+when it costs no more than its worst case, so a run can exceed its budget by at
+most the difference between a request's worst case and its hold, for each
+request in flight.
+
+Read `refusedHoldBasis`: `full` means the hold was a fallback, so the next
+request in a warm conversation would likely hold less; `cached` means it
+already used the cache-read rate, so the budget is simply low for the request.
+`refusedHoldCeilingUsd` is the most the request could ever hold.
+
+## What to do
+
+- `run`: raise `budgetUsd` on the agent, set `allowedOverrunPercent` (see
+  below), or lower `codingProfile.maxOutputTokens`
+  so each request holds less. If the sentence says the allowed overrun wasn't
+  available, raising the overrun will not help: raise the full group's cap, or
+  the lead's `budgetUsd` for a run tree, as the sentence says.
+- `run-tree`: raise `budgetUsd` on the lead agent; its sub-agents draw on it.
+- `day`, `week`, `month`: raise the budget group's cap for that period, or wait
+  for the period to reset. See [Budget group exhausted](budget-group-exhausted.md).
+
+A refused run is not resumed automatically; trigger a new one.
+
+## Allowed overrun
+
+A coding agent's `allowedOverrunPercent` (integer 0 to 100, default 0; set with
+`create_agent` or `update_agent`, same permission as `budgetUsd`) lets a run
+spend up to that percent beyond `budgetUsd`. On a lead agent that delegates,
+it raises its run tree's ceiling for coding sub-agents instead; native runs' own
+model calls are not affected. A delegated run tree's ceiling uses the lead
+agent's percent.
+
+The overrun is permission, not a reservation. A run reserves only its base
+budget (`budgetUsd`, trimmed to the room its budget group and run tree have).
+While it works, it is topped up in steps toward its limit (`budgetUsd` raised
+by the percent), each only from room the group and run tree still have, so
+group caps and run-tree ceilings stay hard. `get_run` shows `budgetBaseUsd`
+(before the overrun), `budgetLimitUsd` (the limit) and `budgetReservedUsd`
+(what has been granted so far). The sentence names what the run had, for
+example "this run's $1.00 budget and its 10% allowed overrun ($1.10 in all)",
+or "$1.00 budget and $0.05 of its 10% allowed overrun ($1.05 in all)".
+
+### Overrun not available
+
+When the group or run tree has no room left for the next top-up, `get_run`
+lists an `overrun_unavailable` event (its `limit` names the full cap or
+`run-tree`), and the run asks for no more, even if room frees up later. The
+proxy then stops it at what it was granted, and the sentence says why, for
+example "Only $0.40 of its 10% allowed overrun was available because the
+"team" group had no more left of its daily cap.", or "Its 10% allowed overrun
+wasn't available because the lead's run tree had nothing left of its $15.00
+budget." when it got none. A top-up takes whatever room is left, so the run is
+refused only once nothing is. A run whose base is smaller than its first model
+request cannot use its overrun: that request is refused before any top-up.
+To continue, raise that cap (or the lead's `budgetUsd`) and continue the draft
+pull request. Runs dispatched before this behaviour have no `budgetLimitUsd`:
+their reservation already included the overrun, and they finish as before.
+
+A worker that finishes is delivered even when it ended over the limit: the run
+is `succeeded` and its status comment says, for example, "Spent $1.04, $0.04
+(4%) over its $1.00 budget, within its allowed overrun."
+
+## Keeping the work of a stopped run
+
+When a run is stopped for budget and its worker ended cleanly, its changes so
+far are pushed as the usual draft pull request (or added to the one it was
+continuing). A new pull request's description starts "Stopped at its budget
+before finishing; the changes so far are in this draft." A continuation leaves
+the existing description alone. The run stays `budget_exhausted` and
+keeps its `budgetEvents`. Mentioning the agent on that pull request (a
+comment mentioning the App), or on the issue it came from (the App on a
+GitHub issue, the service account on a Jira issue), continues the work with
+a fresh budget. The pull
+request is not marked ready automatically until a later continuation finishes.
+
+A new pull request's description has a `### Task` section quoting the request
+that started the run (capped at 4 KiB, mentions defused), so a continuation
+knows what the draft was for. The request text is visible to anyone who can
+read the pull request. While the draft's latest commit is the stopped
+run's push, the automatic review is skipped with a neutral check, "Skipped:
+stopped at its budget", and no fix round starts. Marking the pull request ready
+or pushing a new commit gets a normal review. See
+[Run GitHub code-review agents](../code-review-agents.md#drafts-stopped-at-their-budget).
+
+If a push check refuses (protected paths, diff limits, access) or there are no
+changes, the run ends as above with nothing pushed. So does a continuation of a
+pull request that is no longer a draft (marked ready for review): unfinished
+work is never pushed onto it. A worker that crashed, timed out or was stopped
+is not kept.
+
+## Interrupted replies
+
+When a reply breaks off before its final usage, the proxy settles the request
+at an estimate instead of keeping its whole hold: the usage it already read,
+else the input of the same conversation's previous reply on the same model plus
+the body's growth since then, else the whole body as new input, plus the output
+relayed so far (text, tool-call input, and thinking or reasoning text) counted
+high, and never more than the request's worst case. With nothing relayed, the
+run's median completed reply on that model is used, or no output if there is
+none yet. The previous reply is used only if it started within the 5-minute
+prompt cache window before the request and the body has not shrunk since. New
+input is counted at one token per 3 bytes and priced at the higher of the
+model's input and cache-write rates. A streaming request that failed before the
+provider's response started books no output. The native sandbox gateway follows
+the same rule: a call cut after the response started but before any output (for
+example while the model was thinking) is booked at its input plus the run's
+median completed reply, or its input alone if there is none yet. The request is
+flagged as estimated and counted in the run's cost. If the estimate cannot be
+computed or recorded, the request keeps its full hold. A run can exceed its budget by at
+most the gap between the estimate and the real cost.
+
+## Group holds for delegated runs
+
+A lead and the sub-agents it starts in the same budget group hold the lead's
+budget once, at the lead (its reservation minus what the whole tree has spent),
+decided per period. If the tree's live sub-agent runs hold more than that
+(a coding run can be topped up past the lead's budget by the lead's
+`allowedOverrunPercent`), the lead holds their unspent holds instead. `get_budget_group` reports `heldRuns` for each period.
+
+See also [Troubleshoot budgets and reservations](../troubleshooting/budgets.md).
+
+A coding run refused by the model provider itself because its account is out of
+credit or over a spend limit is a different failure; see
+[errors/coding-stream-quota-exceeded](coding-stream-quota-exceeded.md).

package/help/errors/budget-group-exhausted.md

@@ -14,7 +14,9 @@ has no capacity left for that period after accounting for actual spending and
 active reservations. Wardby refuses the run before model work or a coding
 worker begins.
 
-1. Inspect the budget group's `spentUsd` and `reservedUsd` values.
+1. Inspect the budget group's `spentUsd`, `reservedUsd` and `heldRuns` values.
+   The refused run's `get_run` output lists the cap, spend and holds in
+   `budgetEvents`.
 2. Inspect active and recent runs that use the group.
 3. Wait for the period to reset, cancel unneeded active work, or deliberately
    adjust the agent or group budget.
@@ -23,3 +25,5 @@ worker begins.
 
 Do not raise a limit merely to clear a refusal without confirming the intended
 owner, schedule, and overlapping work. See [Budget troubleshooting](../troubleshooting/budgets.md).
+
+For a coding run stopped mid-run, see the help article `errors/budget-exhausted`.

package/help/errors/coding-provider-not-configured.md

@@ -26,8 +26,8 @@ runs recorded it) when `CODING_WORKER_IMAGE` has since been unset.
   a toolchain with its own image (such as `CODING_WORKER_IMAGE_NODE_PYTHON_3_12`),
   still runs without `CODING_WORKER_IMAGE`.
 - **`coding_provider_not_configured:claude-code`**: `CODING_CLAUDE_WORKER_IMAGE`
-  or `CODING_CLAUDE_TOOL_RUNNER_IMAGE` isn't set. Claude Code needs both, and
-  even an agent's own `workerImageRef` doesn't replace the tool runner.
+  or `CODING_CLAUDE_TOOL_RUNNER_IMAGE` isn't set. Claude Code needs both, even
+  for an agent with its own `workerImageRef`.
 
 This error is about images, not API keys. A missing or invalid model API key
 fails later, when the run calls the model.

package/help/errors/coding-stream-quota-exceeded.md

@@ -0,0 +1,33 @@
+---
+id: errors/coding-stream-quota-exceeded
+title: Coding provider out of credit or over its spend limit
+summary: A coding run's model provider refused it because the account is out of credit or over its spend limit; retrying does not help until the limit is raised.
+audience: operator
+tags: [error, coding-agents, billing, quota, spend-limit, coding_stream_quota_exceeded]
+appliesTo: ">=0.7.0"
+---
+
+# Coding provider out of credit or over its spend limit
+
+`job_coding_stream_quota_exceeded` in a coding run's diagnostic (the
+control-plane log line under the run's `coding_diag_…` id) means the model
+provider refused the run's requests because the provider account behind the
+coding proxy has run out of credit or reached a spend limit. OpenAI reports
+this as `insufficient_quota` or a project spend limit (with HTTP 429, like a
+rate limit); Anthropic as a credit balance that is too low. It is not wardby's
+own run budget: that ends the run as `budget_exhausted` (see
+[errors/budget-exhausted](budget-exhausted.md)).
+
+Retrying does not help: every coding run on that provider account fails the
+same way until the limit changes.
+
+1. Check the provider account the coding proxy uses: the OpenAI project's
+   billing and limits, or the Anthropic organization's credit balance and spend
+   limits.
+2. Add credit or raise the limit there.
+3. Trigger the run again.
+
+The coding proxy's log for the same request (`audit.*` events) shows the
+provider's status. See
+[Coding worker isolation](../../docs/coding-worker-isolation.md) for where the
+diagnostic and proxy logs live.

package/help/errors/coding-workspace-disk-exceeds-limit.md

@@ -0,0 +1,39 @@
+---
+id: errors/coding-workspace-disk-exceeds-limit
+title: Coding workspace larger than the deployment allows
+summary: create_agent and update_agent refuse workspaceDiskMb above CODING_MAX_DISK_MB; already-saved agents above the cap fail at run start.
+audience: operator
+tags: [error, coding-agents, workspace, disk, workspaceDiskMb, CODING_MAX_DISK_MB, create_agent, update_agent]
+appliesTo: ">=0.2.0"
+---
+
+# Coding workspace larger than the deployment allows
+
+`coding_workspace_disk_exceeds_limit` (on the run record as
+`coding_failure_workspace:<id>`) means the agent's
+`codingProfile.workspaceDiskMb` is larger than the deployment's
+`CODING_MAX_DISK_MB`. The run fails before a worker starts and costs nothing.
+When `CODING_MAX_DISK_MB` is unset, the limit is `CODING_DISK_MB` (2048 MiB by
+default).
+
+`create_agent` also checks this limit at save time. `update_agent` checks it
+when setting a numeric `workspaceDiskMb` or changing an agent's kind to coding.
+You can still make unrelated edits to an already-saved over-cap coding agent,
+lower its size to within the cap, or set it to `null`.
+The error names the requested size in MiB,
+the effective cap in MiB, and `CODING_MAX_DISK_MB`. A size exactly at the cap
+is allowed; `null` uses the deployment default. Agents already saved above
+the cap still fail at run time, including if the operator lowers the cap.
+
+Fix it one of two ways:
+
+1. Lower the agent's size to the limit or below with `update_agent`:
+   `{ "id": "<agent id>", "codingProfile": { "workspaceDiskMb": 2048 } }`.
+2. Or raise the deployment's `CODING_MAX_DISK_MB` on the control plane and
+   restart it. On GKE, deploy with `CODING_MAX_DISK_MB=8192 deploy/gke/up.sh`
+   (see the GKE deploy guide). On GKE Autopilot the value must leave room for the
+   worker's 1 GiB inside the 10 GiB pod ephemeral-storage ceiling.
+
+A run that installs more than its workspace holds is evicted mid-run instead;
+that needs a larger `workspaceDiskMb`, within this limit. See
+[Build a worker image](../build-worker-image.md).

package/help/errors/continuation-closed.md

@@ -19,15 +19,13 @@ an open pull request carrying that run's marker for the branch. Nothing from
 the run was pushed either way, so the pull request (and anyone who merged or
 closed it) is unaffected.
 
-Two things besides an actual merge or close can also make the check come back
-"no longer found," since it looks for an **open** pull request with that
-exact base branch and the run's hidden marker still in its description:
-
-- the pull request's base branch was retargeted to something other than the
-  one the run recorded;
-- the hidden `<!-- wardby:<run-id> -->` marker was removed or edited out of
-  the pull request's description (wardby relies on it to find its own PR;
-  nothing else identifies it).
+One thing besides an actual merge or close can also make the check come back
+"no longer found," since it looks for an **open** pull request from the run's
+branch with the run's hidden marker still in its description: the hidden
+`<!-- wardby:<run-id> -->` marker was removed or edited out of the pull
+request's description (wardby relies on it to find its own PR; nothing else
+identifies it). Retargeting the pull request to another base branch does not
+close it for wardby: the follow-up continues it and merges the new base.
 
 When the pull request was merged or closed **before the run started**, the
 check before cloning catches it: the run's status is `refused`, and nothing
@@ -66,9 +64,8 @@ sub-run instead, without the error code:
 If the change is still needed, ask again without `continuePriorRun` (or
 without naming the closed pull request). The new run opens a fresh pull
 request from the repository's default branch. If the check is wrongly
-finding nothing when the pull request is in fact open, check that its base
-branch still matches what wardby recorded and that the `<!-- wardby:... -->`
-marker is still in its description, unedited.
+finding nothing when the pull request is in fact open, check that the
+`<!-- wardby:... -->` marker is still in its description, unedited.
 
 Related: [Run GitHub code-review agents](../code-review-agents.md),
 [Automatic review fix rounds](../review-fix-rounds.md),

package/help/errors/database-auth-invalid.md

@@ -0,0 +1,28 @@
+---
+id: errors/database-auth-invalid
+title: Database authentication is misconfigured
+summary: DATABASE_AUTH or the database URL does not meet the rules for the chosen authentication mode, so the process refused to start.
+audience: operator
+tags: [error, database, rds, iam, deploy, database_auth_invalid, rds-iam]
+appliesTo: ">=0.5.4"
+---
+
+# Database authentication is misconfigured
+
+`database_auth_invalid` means `DATABASE_AUTH` is not valid, or `DATABASE_AUTH=rds-iam` is set and `DATABASE_URL` (or `DBOS_SYSTEM_DATABASE_URL`) does not meet its rules. Wardby checks this when the process starts, so every `wardby` command fails with it, not only the server. The message names the problem; it never includes the URL, credentials, or parameter values.
+
+`DATABASE_AUTH` is `password` (the default; the password is in `DATABASE_URL`) or `rds-iam` (Amazon RDS IAM database authentication, where the password is a short-lived token signed with the workload's AWS credentials). Any other value is refused.
+
+Under `rds-iam` check each of these:
+
+1. `DATABASE_URL` starts with `postgresql://` or `postgres://`, has a user, and has no password.
+2. `AWS_REGION` or `DATABASE_AWS_REGION` is set (`DATABASE_AWS_REGION` wins).
+3. The URL has `sslmode=verify-full` and `sslrootcert=<path to the RDS CA bundle>`, and that file exists and is readable in the container. An unreadable file is reported with its error code, such as `ENOENT` for a missing file or `EACCES` for one the process cannot read.
+4. Only supported parameters are present: `sslmode`, `sslrootcert`, `application_name`, `fallback_application_name`, `options`, `client_encoding`, `statement_timeout`, `lock_timeout`, `idle_in_transaction_session_timeout`, `query_timeout`, `connection_limit`, `pool_timeout`, `connect_timeout`, `schema`. Anything else (`sslcert`, `sslkey`, `ssl`, `host`, `pgbouncer`, ...) is refused by name; remove it.
+5. User and database names in the URL are correctly percent-encoded.
+
+If the code is not raised but connections are rejected, the configuration is valid and the problem is on the AWS side: the workload's IAM role needs `rds-db:connect` for the database user, and that user needs the `rds_iam` role in the database.
+
+`prisma migrate deploy` and `dbos schema` take a URL and do not follow `DATABASE_AUTH`. Mint a short-lived URL with the migration image's helper: `DATABASE_URL="$(node scripts/rds-iam-url.mjs)" && export DATABASE_URL && npm run prisma:migrate && npm run dbos:migrate`. It reports the same `database_auth_invalid` errors for the checks it shares. `wardby quickstart` does not support `rds-iam`.
+
+See [Database authentication](../../docs/security-deployment.md#database-authentication).