@wardby/cli 0.6.0 → 0.8.0

package/docs/debugging-agent-runs.md

@@ -0,0 +1,242 @@
+# Debugging an agent run
+
+You can triage most agent problems yourself with the Wardby MCP and GitHub. Only
+some failures need someone with server logs.
+
+## Start here
+
+Find the run with `list_runs`, then read it with `get_run`. Then go to the section
+for what you see.
+
+```mermaid
+flowchart TD
+    start["Find the run: list_runs, then get_run"]
+    start --> none["No run started<br/>Check the repo links and triggers"]
+    start --> failed["Failed or refused<br/>Read failureCategory and the final text"]
+    start --> nopr["Succeeded, no PR<br/>Read outcome in codingResult"]
+    start --> missed["PR, but it missed<br/>Read the diff and the review"]
+    none --> nopull["See: No pull request appeared"]
+    nopr --> nopull
+    failed --> runfailed["See: The run failed"]
+    missed --> didntfix["See: The agent didn't fix what you expected"]
+    runfailed -- still unexplained --> operator["See: When you need an operator"]
+    nopr -- why it decided that --> transcript["See: Run transcripts"]
+    missed -- why it decided that --> transcript
+```
+
+## What you can use without server access
+
+These all work through the Wardby MCP or GitHub, and you only see runs you have
+access to.
+
+| Tool                                                                                                            | What it tells you                                                                                                                                                                                                                                                                                                                                                                        |
+| --------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `list_runs` (agent id, optional `status`)                                                                       | An agent's recent runs: status, trigger, cost, turns, final text, and which run started it                                                                                                                                                                                                                                                                                               |
+| `list_runs` without an agent id (from 0.8.0)                                                                    | Your runs across every agent, each with `agentName`; no final text. `active: true` or `since` (ISO time) narrow it                                                                                                                                                                                                                                                                       |
+| `get_run` (run id)                                                                                              | One run in full, adding `failureCategory`, `diagnosticId`, `codingResult.outcome`, `request`, budget events and package refusals                                                                                                                                                                                                                                                         |
+| `triggerInfo`, `outcomes`, `serviceReadiness` (from 0.8.0, on `get_run` and on `list_runs` without an agent id) | What started the run (the pull request and whether it was a review or a mention, the issue, the schedule or the lead), what it produced (pull requests opened, comments, checks) and each service's state, in the admin viewer API's shapes. Follow a pull request from the run that opened it through its review and fix rounds by matching one run's outcome to the next run's trigger |
+| `get_agent`, `list_repositories`, `list_subagents`                                                              | How the agent is set up: prompt, model, budget, repo links and triggers, sub-agents                                                                                                                                                                                                                                                                                                      |
+| `search_help`, `get_help_article`                                                                               | Help for an error code or failure category                                                                                                                                                                                                                                                                                                                                               |
+| `cost_report`                                                                                                   | What runs cost, when spend is the question                                                                                                                                                                                                                                                                                                                                               |
+| `get_run_transcript`                                                                                            | The run's conversation, for Claude Code runs with `sessionResume` on                                                                                                                                                                                                                                                                                                                     |
+| `stop_run`                                                                                                      | Stops a run that's stuck                                                                                                                                                                                                                                                                                                                                                                 |
+| The PR, issue or Jira ticket                                                                                    | The diff, CI, the review, and the comment the run posted                                                                                                                                                                                                                                                                                                                                 |
+
+From Wardby 0.8.0, `list_agents` with `summary: true` returns each agent's id,
+name, kind, model, coding provider, budget, owner and your access, without the
+prompt and coding profile. Admins (the `admin` role, `admin:view` scope) can add
+`all: true` to `list_agents` or `list_runs` to see every agent or run in the
+deployment.
+
+Admins can also use the [Wardby Viewer](viewer-api.md), a desktop app showing runs,
+sub-agent trees, triggers and outcomes for every owner.
+
+## The run failed
+
+Call `get_run` and read `status`, `error`, `failureCategory` and the final text. A
+failed coding run's `error` looks like
+`coding_failure_repo_access:coding_diag_…`: the middle part is the stage that
+failed, and the ID is what an operator searches for.
+
+| What you see                    | Likely cause                                              | What to do                                                           |
+| ------------------------------- | --------------------------------------------------------- | -------------------------------------------------------------------- |
+| `repo_access`                   | The agent's owner lost access, or a repo link was removed | Check `list_repositories` and the owner's GitHub access              |
+| `repo_access_unavailable`       | GitHub couldn't be reached reliably                       | Try again later                                                      |
+| `workspace`                     | Often a workspace disk size above the server's limit      | Lower `workspaceDiskMb`, or ask an operator to raise the limit       |
+| `executor` while running checks | The run may have run out of memory, among other causes    | Reduce what the checks run; send the ID to an operator if it repeats |
+| `merge_conflict`                | The branch conflicts with its target                      | Read the `merge-conflict` help article                               |
+| `service_unready`               | A service the run needs, such as Postgres, didn't start   | Check `.wardby/services.yaml` and the agent's `services`             |
+| `coding_turn_limit`             | The run used all its turns                                | Raise `maxTurns` or narrow the task                                  |
+| `budgetEvents`, or `refused`    | The run, run tree or budget group ran out of money        | Raise the budget or check spend with `cost_report`                   |
+| `packageRefusals`               | A package isn't allowed or is too new                     | Add it to `packageAllowlist`, or wait out the release age limit      |
+
+For anything else, call `search_help` with the code or category. If that doesn't
+explain it, see [When you need an operator](#when-you-need-an-operator).
+
+**Prompt for Claude Code:**
+
+```
+Wardby run <run id> failed. Why, and what can I do about it?
+If it needs server logs, give me the diagnosticId.
+```
+
+## No pull request appeared
+
+First find out whether a run started, using `list_runs` on the lead or router,
+then on the coding agent.
+
+| What `list_runs` shows           | What it means                                                                                              | What to do                                                                                      |
+| -------------------------------- | ---------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
+| No new run                       | The request never started anything: nothing is linked with that trigger, or the person doesn't have access | Check `list_repositories` on the router. Only one agent per repo can hold the `mention` trigger |
+| Lead ran, no coding run under it | The lead chose not to delegate                                                                             | Read the lead's final text, which should say why                                                |
+| Coding run failed or was refused | See [The run failed](#the-run-failed)                                                                      |                                                                                                 |
+| Coding run succeeded, with no PR | Check `codingResult.outcome` in `get_run`                                                                  | See below                                                                                       |
+
+| `outcome`          | Meaning                                                                                                                                                 |
+| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `no_changes`       | The agent changed nothing. Its final text should say why: the request may be vague or already done. On Claude Code, the transcript shows how it decided |
+| `budget_exhausted` | It ran out of money before finishing. Raise the budget or narrow the task                                                                               |
+| `branch_pushed`    | A local repo: there's a branch, and no PR by design                                                                                                     |
+
+A mention can also get no reply at all. If a server sets
+`WARDBY_MENTION_EXPLANATIONS=false`, a mention that starts nothing stays silent. An
+operator can look for `mention ignored` or `mention declined` in the server log.
+
+**Prompt for Claude Code:**
+
+```
+I asked Wardby agent <agent name> to <request> in <your-org/your-repo>,
+and no pull request appeared. Why?
+```
+
+## The agent didn't fix what you expected
+
+The run succeeded, so most of what you need is on the PR and the run, with no
+server logs involved. Check these in order:
+
+1. **The diff:** what actually changed.
+2. **The PR description:** the run's final text, including what it says it tested.
+   If it says the tests weren't run, trust CI.
+3. **The review:** whether your review check caught the miss.
+4. **The request:** whether the issue or mention said what you meant.
+5. **The task the agent received:** `request` in `get_run` (from Wardby 0.8.0). On a
+   delegated run, this shows what the lead told the coding agent.
+6. **The transcript:** why it decided that. This is only for Claude Code runs with
+   `sessionResume` on.
+
+The usual causes are a vague request, a lead that dropped something when it rewrote
+the request, a repo rule missing from `AGENTS.md`, or a fix aimed at the symptom it
+was told about.
+
+**To fix it,** comment on the PR, mentioning the App, with what's wrong. The
+follow-up pushes to the same branch and PR, so you don't start again. If the same
+miss keeps happening, add the rule to `AGENTS.md` so future runs follow it.
+
+**Prompt for Claude Code:**
+
+```
+PR <PR URL> from Wardby didn't do what I expected: <what you expected>.
+Where did it go wrong? Draft a follow-up comment, but don't post it.
+```
+
+## Run transcripts
+
+A transcript is the run's conversation: the messages, the tool calls and their
+results, and the files and skills the agent loaded. You can get it yourself,
+without server access, on Claude Code agents.
+
+**To turn it on,** set `codingProfile.sessionResume` to `true` with `update_agent`.
+It's off by default and has to be turned on for each agent. It only applies to runs
+started afterward, so an earlier failure has no transcript. The server also needs a
+blob store, set in the environment of the process that launches coding runs:
+`BLOB_STORE` set to `local`, `gcs` or `s3`, plus `BLOB_STORE_LOCAL_DIR`,
+`BLOB_STORE_GCS_BUCKET`, or `BLOB_STORE_S3_BUCKET` and `BLOB_STORE_S3_REGION`.
+Without `BLOB_STORE`, no transcript is stored.
+
+**To read one,** call `get_run_transcript` with the run ID. It returns a download
+link that works for 10 minutes and a command. Run `npx @wardby/cli@latest resume -`
+in a clone of the repo and paste the link, which also lets you continue the session
+on your machine.
+
+| Transcripts        |                                                                                |
+| ------------------ | ------------------------------------------------------------------------------ |
+| **Who can read**   | Anyone who can read the run: the person who triggered it and the agent's owner |
+| **Stored for**     | Runs that finish, fail, or stop at their budget                                |
+| **Not stored for** | Runs stopped with `stop_run` or ended by their timeout                         |
+| **Removed**        | Thinking blocks, the system prompt, the worker's environment, account details  |
+| **Redacted**       | The agent's attached secrets and the run's own tokens                          |
+| **Not available**  | On Codex agents                                                                |
+
+Redaction removes secret values, not other sensitive text, so a transcript includes
+what the agent saw in the repo and in command output. Turn this on only for agents
+whose runs you're comfortable sharing with everyone who can read them.
+
+**Prompts for Claude Code:**
+
+To turn it on:
+
+```
+Turn on transcripts for Wardby agent <agent name>.
+```
+
+To get one:
+
+```
+Get me the transcript for Wardby run <run id>.
+```
+
+See [Resume Claude Code runs locally](../help/resume-runs-locally.md) for more.
+
+## When you need an operator
+
+Ask someone with server access when `get_run` and help don't explain a failure. The
+specific reason for a failed coding run is written only to the server log, under
+its `diagnosticId`.
+
+**Send them:** the run ID, the `diagnosticId`, the repo, and what you expected to
+happen.
+
+**What the operator can do:**
+
+1. Search the control-plane log for the `diagnosticId`, which holds the real reason.
+2. If that isn't enough, turn on the
+   [debug trace](coding-worker-isolation.md#debug-trace) with `debugTraceMinutes`
+   and reproduce the failure.
+
+The debug trace won't help you directly. Only admins can turn it on, it only covers
+runs started afterward, and its output goes to the run's worker log, which you can't
+read. It can include prompts and repo content, so it isn't exposed through MCP.
+
+**What the trace holds:** the agent's whole stream, the commands it ran with their
+output (Claude Code), final usage and duration, and stream failures in full.
+Token-shaped values are redacted. It stops at 2 MiB per run by default.
+
+**What MCP gives you:** the trace itself isn't available through MCP. You can only
+turn it on or off with `update_agent` (admin role), see when it expires with
+`get_agent` (`debugTraceUntil`), and see whether a run was traced with `get_run`
+(`debugTrace: true`).
+
+**Prompt to try first.** Paste this into any client connected to the Wardby MCP:
+
+```
+Wardby run <run id> didn't do what I expected. What happened, and what can I do?
+If it needs server logs, give me the diagnosticId.
+```
+
+## Known gaps
+
+Two things can't be answered without server access yet.
+
+| Gap                                     | Why it matters                                                       | What you can do                                                     |
+| --------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------- |
+| The specific reason a coding run failed | You get the stage, such as `executor`, but not why                   | Send the `diagnosticId` to an operator                              |
+| Why a mention got no reply              | With explanations off, a declined mention leaves no reply and no run | Turn explanations on where that's acceptable: `true` is the default |
+
+**To turn explanations on,** an operator removes
+`WARDBY_MENTION_EXPLANATIONS=false` from the server's settings, or sets it to
+`true`, then redeploys. Once on, the GitHub App replies to say why a mention started
+nothing. On a public repo, anyone who can read that comment can see what is or isn't
+linked there.
+
+Run transcripts are limited to Claude Code runs with `sessionResume` on, so they
+don't help on Codex agents.

package/docs/getting-started-gke.md

@@ -50,6 +50,7 @@ Install and authenticate:
 - `kubectl`
 - Helm 3 or later (installs External Secrets Operator)
 - Node.js 24 and npm
+- `curl`, `openssl` and `base64` (the database grants step uses them)
 
 You also need:
 
@@ -200,7 +201,9 @@ refuses to start with either one set and no token.
 `WARDBY_GITHUB_ALLOWED_ACCOUNTS` (optional, not a secret) adds GitHub accounts
 the App serves besides its owner. Pass it to `up.sh` on every run, for example
 `WARDBY_GITHUB_ALLOWED_ACCOUNTS=acme-labs WARDBY_HOSTNAME=app.example.com deploy/gke/up.sh`;
-a run without it serves only the App's owner. See
+a run without it serves only the App's owner. `up.sh` gives the same value to
+the control plane and to the native sandbox gateway, which runs sandboxed
+agents' repository tools. See
 [Which accounts the App serves](code-review-agents.md#which-accounts-the-app-serves-wardby_github_allowed_accounts).
 
 `WARDBY_MOBILE_ENABLED=true` (optional, not a secret) lets the Wardby mobile app
@@ -210,6 +213,11 @@ sign-in off. It works for normal and low-power deploys. White-label builds that
 use their own redirect URI add `WARDBY_MOBILE_REDIRECT_URI` to the control
 plane's `env` themselves. See the help article `mobile-app`.
 
+`WARDBY_MENTION_EXPLANATIONS=false` (optional, not a secret) stops the GitHub
+App replying to say why a mention started nothing; pass it to `up.sh` on every
+run, the same way. See
+[What a linked review agent does](code-review-agents.md#what-a-linked-review-agent-does).
+
 Generate `SECRET_APP_KEY` with:
 
 ```sh
@@ -315,10 +323,9 @@ Allow DNS and the managed certificate to converge before treating an HTTPS
 failure as an application failure.
 
 On a brand-new project, run `terraform apply` and then
-`bootstrap-database-iam.sh` before `up.sh`: the bootstrap has to grant the
-migrator before any migration can run. See "Database login" below for the
-exact order, including the database check `up.sh` doesn't pass until a second
-bootstrap run has granted the coding proxy too.
+`bootstrap-database-iam.sh` once before the first `up.sh`: the bootstrap has to
+grant the migrator before any migration can run. See "Database login" below
+for the exact order.
 
 ### Database login
 
@@ -327,20 +334,19 @@ through the Cloud SQL Auth Proxy, via Workload Identity from one Kubernetes
 service account. What each may do inside the database comes from a `NOLOGIN`
 group role in `deploy/gke/database-grants.sql`:
 
-| Workload       | Google service account   | Kubernetes service account | May do                                                                                                                                                                                          |
-| -------------- | ------------------------ | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| Control plane  | `<name_prefix>-app`      | `wardby-control-plane`     | `wardby_app`: read/write every table, including the durable executor's in schema `dbos`; never change a schema                                                                                  |
-| Coding proxy   | `<name_prefix>-proxy`    | `wardby-coding-proxy`      | `wardby_proxy`: only its budget ledger — `CodingProxySession`/`CodingProxyRequest`/`RunModelUsage`, plus update `tokensIn`, `tokensOut`, `costUsd` and `turns` on `Run`, and read only its `id` |
-| Native gateway | `<name_prefix>-gateway`  | `wardby-native-gateway`    | Member of `wardby_app`: the same data access as the control plane, under its own login (see [Native sandbox](#native-sandbox))                                                                  |
-| Migrations     | `<name_prefix>-migrator` | `wardby-migrator`          | Acts as the table owner (`SET ROLE`), so `prisma migrate deploy` and `dbos schema` can alter and create tables                                                                                  |
+| Workload       | Google service account   | Kubernetes service account | May do                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
+| -------------- | ------------------------ | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Control plane  | `<name_prefix>-app`      | `wardby-control-plane`     | `wardby_app`: read/write every table, including the durable executor's in schema `dbos`; never change a schema                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
+| Coding proxy   | `<name_prefix>-proxy`    | `wardby-coding-proxy`      | `wardby_proxy`: its budget ledger (`CodingProxySession`/`CodingProxyRequest`/`RunModelUsage`: read, insert, update), the package registry's tables (`RegistryAllowance`, `RegistryFetch`, `RegistryAuditRecord`, `RegistryVersionFact`, `RegistryApprovedVersion`, `RegistryPlanRefusal`: read and insert only), the dependency cache's `DependencySnapshot` (read, insert, update, delete), update `tokensIn`, `tokensOut`, `costUsd` and `turns` on `Run` and read only its `id`, and read only the `runId`, `packageAllowlist`, `packagePolicy`, `repository`, `workerImage`, `toolImage`, `dependencyCache` and `packageAuditMode` columns of `CodingRun` |
+| Native gateway | `<name_prefix>-gateway`  | `wardby-native-gateway`    | Member of `wardby_app`: the same data access as the control plane, under its own login (see [Native sandbox](#native-sandbox))                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
+| Migrations     | `<name_prefix>-migrator` | `wardby-migrator`          | Acts as the table owner (`SET ROLE`), so `prisma migrate deploy` and `dbos schema` can alter and create tables                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
 
 `deploy/gke/bootstrap-database-iam.sh` applies the grants as the built-in
-owner, from a short-lived Job inside the cluster. Run it whenever
-`database-grants.sql` changes, before deploying the release that needs the new
-grants (`bootstrap-database-iam.sh --check`, then `bootstrap-database-iam.sh`,
-then `up.sh`), and as part of the orders below. A release deployed ahead of its
-grants fails with "permission denied" wherever it uses one it doesn't have yet.
-The grants run in one transaction, so a statement the database refuses leaves
+owner, from a short-lived Job inside the cluster. `up.sh` runs it on every
+deploy, after the migrations and before the rollout (see
+[Upgrading an existing deployment](#upgrading-an-existing-deployment)); you
+run it by hand only on a brand-new project, below, or with `--check`. The
+grants run in one transaction, so a statement the database refuses leaves
 nothing applied.
 
 The built-in owner is not a Terraform resource: `bootstrap-database-iam.sh`
@@ -349,119 +355,28 @@ SQL Admin API, uses that password once to apply the grants, and resets it to
 a value nobody holds on every exit, including a failed grant. No password is
 ever stored, printed, or passed as a process argument.
 
-A deployment that already runs and then gains the native gateway identity from
-`terraform apply` runs `deploy/gke/bootstrap-database-iam.sh` once, so the
-gateway's user gets its `wardby_app` membership, and then `deploy/gke/up.sh`.
-`up.sh`'s database check also tests the native gateway and says so if the
-grant is missing.
-
 A brand-new project runs, in this order:
 
 1. `terraform -chdir=deploy/gke apply` — creates the cluster, the instance
    and the four IAM database users. Run it yourself rather than through
    `up.sh`, which would go on to the migrations before the migrator has its
    grants.
-2. `deploy/gke/bootstrap-database-iam.sh` (default mode) — fetches the
-   cluster's kubectl credentials if you don't have them yet, creates the
-   owner, and applies the migrator's (and the app's) grants. The coding
-   proxy's ledger tables don't exist yet, so its grants are skipped: expected.
-3. `deploy/gke/up.sh` — applies the migrations and rolls out, then stops at
-   the database check: the coding proxy's grants are on tables that did not
-   exist in step 2.
-4. `deploy/gke/bootstrap-database-iam.sh` again — now that the tables exist,
-   also applies the coding proxy's ledger grants.
-5. `deploy/gke/up.sh` — rolls out again and passes every check.
-
-Two bootstrap runs, not one: a grant on a named table can't apply before the
-migration that creates that table has run.
-
-The same applies to every later release that adds a table the coding proxy
-uses: run `bootstrap-database-iam.sh` again after its migration. `up.sh`
-checks this for you. After the rollout it asks the database, as the proxy's
-own role, for every privilege `database-grants.sql` gives `wardby_proxy`
-(`deploy/gke/proxy-grant-checks.mjs`), and stops with the list of missing
-grants instead of letting the proxy fail registry requests with "permission
-denied".
-
-After upgrading an existing deployment to a release with cost attribution,
-re-run `bootstrap-database-iam.sh` so the coding proxy can record per-model
-usage for coding runs (used by the `cost_report` tool's `model` grouping).
-Until then coding runs still work, but their per-model breakdown isn't
-recorded.
+2. `deploy/gke/bootstrap-database-iam.sh` — fetches the cluster's kubectl
+   credentials if you don't have them yet, creates the owner, and applies the
+   migrator's (and the app's) grants. Grants on tables that don't exist yet
+   are skipped: expected.
+3. `deploy/gke/up.sh` — applies the migrations, then the grants again (now
+   that every table exists), then rolls out and passes every check.
 
 `bootstrap-database-iam.sh --check` runs the grants inside a transaction that
 is then rolled back, and reports success or the exact statement the database
-refused, without changing anything. Run it before the real run on a live
-deployment.
+refused, without changing any grant. Use it to see why a grants step failed.
 
 `connector_enforcement` (the Cloud SQL instance setting) defaults to
 `REQUIRED`: it refuses any connection that does not come through the Auth
 Proxy or a Cloud SQL connector, and with port 5432 to the instance closed in
 every NetworkPolicy, there's no route left for a plain Postgres connection to
-even attempt. Leave it at the default; an older deployment still on password
-login moves off it on an earlier revision of this module, below.
-
-### Moving an older deployment
-
-A deployment still on password login — from before this module retired it —
-can't adopt this revision directly: this revision no longer syncs the
-password `DATABASE_URL`, closes port 5432 and refuses non-Auth-Proxy
-connections, which would cut off pods still logging in with the password.
-It moves over in stages, because real pods are already serving traffic
-throughout.
-
-**1–2. Cut the running pods to IAM login, on the revision before this one.**
-Check out the revision of this module that added IAM login, before the
-password was retired: the merge of pull request #83 into `main`.
-`git log --oneline --first-parent main -- deploy/gke` lists the merges that
-changed this module, newest first; it's the one titled
-`Merge pull request #83 ...`, just below the merge that retired the password
-(`git log --oneline --merges --grep='#83' main` finds it directly). Then
-`git checkout <that commit>` and follow that revision's own
-`docs/getting-started-gke.md`, "Database login", and its order for "an
-existing deployment still on password login": `bootstrap-database-iam.sh
---password-from-stdin --check`, the same without `--check`, then `up.sh`.
-That rolls out password-less pods while the password still works as a
-fallback, and proves the control plane and the coding proxy each read the
-database through their own IAM login. That revision's
-`connector_enforcement` default is still `NOT_REQUIRED`, so nothing that
-still uses the password is cut off. Then return to this revision
-(`git checkout main`) for the last stage.
-
-**3. Retire the password**, once every pod speaks IAM login and this change
-has merged to `main`:
-
-1. Confirm nothing still uses the password: both Deployments' pods log in as
-   IAM users, and no pod has a password `DATABASE_URL` in its effective env.
-2. Delete the `database-url` Secret Manager secret:
-   `gcloud secrets delete <name_prefix>-database-url --quiet` — this module no
-   longer creates or reads it. Deleting it by hand first, before the next
-   step, is what lets Terraform drop it from state instead of trying to
-   destroy it.
-3. Remove any `connector_enforcement` line from `terraform.tfvars` (or set
-   it to `REQUIRED`), so the new default takes effect. Then `terraform plan`:
-   expect the generated password destroyed, the `database-url` secret and
-   its IAM binding gone from state (already deleted by hand), the old
-   password-login user forgotten (not destroyed), `connector_enforcement`
-   moving to `REQUIRED` on the instance (updated in place, not replaced),
-   and no other destroy. Review
-   the plan, then apply only once it matches that. If apply still fails
-   trying to destroy the `database-url` secret, the secret wasn't actually
-   deleted in step 2 — delete it and re-apply. Never turn off
-   `secrets_deletion_protection` to get past that error: it unprotects every
-   other secret in the same set, including `SECRET_APP_KEY` and the auth
-   keys, and losing `SECRET_APP_KEY` makes every credential already stored
-   in the database unreadable.
-4. `deploy/gke/up.sh` — new ExternalSecrets carry no `DATABASE_URL`,
-   NetworkPolicies no longer allow 5432, migrations, rollout, and the
-   database and endpoint checks.
-5. `deploy/gke/bootstrap-database-iam.sh` (default mode) — the guard now
-   passes, since the Secret no longer has a `DATABASE_URL`: it sets a
-   one-time owner password, reapplies the grants, and resets the password to
-   a value nobody holds.
-6. Verify: a direct password connection is refused (NetworkPolicy blocks
-   5432, and the instance refuses it too), the pods still log in through
-   IAM, and the Terraform state holds no database password.
+even attempt. Leave it at the default.
 
 ## 7. Verify
 
@@ -486,9 +401,10 @@ server answers `415`, because it checks the content type before the token.
 Create the first self-hosted login credential. It is printed once.
 `--role admin` makes this operator account an admin. Only admins can use all
 the privileged operations: `make_owner`, BYO `workerImageRef`, package
-approval, service catalog changes, and model catalog changes. A
-`package-approver` can approve packages only, a `service-manager` can change
-the service catalog only, and a `model-manager` can change the model catalog
+approval, service catalog changes, model catalog changes, and host providers
+(`hosts:admin`). A `package-approver` can approve packages only, a
+`service-manager` can change the service catalog only, a `model-manager` can
+change the model catalog only, and a `host-manager` can manage host providers
 only. Users created without `--role` have no roles. See
 [roles and privileged operations](security-deployment.md#roles-and-privileged-operations).
 
@@ -552,6 +468,38 @@ roll; if it fails, `up.sh` prints the `migrate` and `cloud-sql-proxy` container
 logs (and, if those are empty or the Job timed out, the Job's and pod's
 events), then stops.
 
+### Upgrading an existing deployment
+
+To upgrade, check out the new release and re-run `deploy/gke/up.sh`. No
+manual database step is needed: `up.sh` applies the migrations, then the
+database grants (`deploy/gke/bootstrap-database-iam.sh`, which applies
+`deploy/gke/database-grants.sql`), and only then rolls out the new pods. A
+release that adds a table, a column, or a workload that needs a new grant
+therefore never runs without it.
+
+- **If the migrations fail**, `up.sh` stops before the grants and the
+  rollout; the running version is untouched.
+- **If the grants fail**, `up.sh` stops before the rollout with a non-zero
+  exit; the running version keeps serving. The migrations already applied
+  are additive, so the running version tolerates them. Run
+  `deploy/gke/bootstrap-database-iam.sh --check` to see the exact statement
+  the database refused without changing anything, fix the cause, and re-run
+  `up.sh`.
+- **After the rollout**, `up.sh` checks that each new pod can read the
+  database, and asks the database, as the coding proxy's own role, for every
+  privilege `database-grants.sql` gives it
+  (`deploy/gke/proxy-grant-checks.mjs`). A missing grant stops the deploy
+  with a non-zero exit and the list of what's missing, rather than letting
+  the proxy fail registry requests with "permission denied".
+
+Never run two `up.sh` deploys against the same project at once: both would
+run migrations, set and reset the database owner's password, and roll out
+over each other. Nothing locks against it, so wait for one to finish.
+
+You run `bootstrap-database-iam.sh` by hand only on a brand-new project,
+between `terraform apply` and the first `up.sh` (see
+[Database login](#database-login)), or as `--check` for a dry run.
+
 ### Durable executor
 
 The control plane runs native runs on the durable executor (`EXECUTOR=dbos`):
@@ -576,24 +524,33 @@ the step that was in flight runs again. Coding runs are Kubernetes Jobs and are
   `up.sh` re-run whose source did not change, runs continue. When you deploy
   new code, the old pod first **drains**: after its 30-second `preStop` it
   stops the scheduler and waits for the runs it is executing to finish, up to
-  `SHUTDOWN_DRAIN_SECONDS` (default 600), while the new pod serves traffic.
-  An idle pod exits at once. A run still going after that limit is marked
-  `lost`; if an `@`-mention started it, its status comment on GitHub says the
-  request was interrupted and should be repeated. The manifest's
-  `terminationGracePeriodSeconds` (720) must stay above the drain limit plus
-  the `preStop` delay.
+  `SHUTDOWN_DRAIN_SECONDS` (540 in the GKE manifest), while the new pod
+  serves traffic. Open HTTP connections, such as a connected MCP client's
+  stream, get 10 seconds to finish and are then closed; clients reconnect to
+  the new pod.
+  An idle pod exits at once. A native run that is waiting in a `delegate_to_*`
+  call on a coding sub-agent or a sandbox-mode native sub-agent is taken over
+  by the new version: it continues when the sub-agent finishes, without
+  repeating the turns it already took. A run that was mid-turn, waiting on
+  another tool, or waiting on a control-plane native sub-agent is marked
+  `lost` rather than taken over, as is a taken-over run whose recorded steps
+  no longer match the new code, and a run still going after the drain limit.
+  The sub-agent runs a lost run started are stopped (reason
+  `parent run lost`). If an `@`-mention started a lost run, its status comment
+  on GitHub says the request was interrupted and should be repeated. The
+  manifest's `terminationGracePeriodSeconds` (600) must stay above the drain
+  limit plus the `preStop` delay. Autopilot does not allow more than 600: a
+  larger value is lowered to 600, so to change the budget, lower
+  `SHUTDOWN_DRAIN_SECONDS` rather than raising the grace period.
 - **Data at rest and retention.** `dbos.operation_outputs` holds every step's
   output — prompts, model responses and full tool results — with no retention
   limit. Pruning finished workflows is the operator's job; see
   [Durable executor](security-deployment.md#durable-executor) for what is
   stored and how to prune it with `DBOS.deleteWorkflow`.
 
-**Enabling it on an existing deployment.** The grants changed, so apply them
-before deploying: run `deploy/gke/bootstrap-database-iam.sh --check`, then
-`deploy/gke/bootstrap-database-iam.sh`, then `deploy/gke/up.sh`. Without the
-grants the new control-plane pod crash-loops with "permission denied" while
-the old one keeps serving, and `up.sh` stops at the rollout. On a brand-new
-project the order under "Database login" already covers it.
+**Enabling it on an existing deployment.** Re-run `deploy/gke/up.sh`: it
+applies the durable executor's grants before the rollout (see
+[Upgrading an existing deployment](#upgrading-an-existing-deployment)).
 
 **Switching back** to the in-process executor: set `EXECUTOR` to `in-process`
 in `deploy/kind-coding/manifests/overlays/gke-autopilot/control-plane.yaml`
@@ -629,7 +586,7 @@ delete the class and re-run `up.sh` to change them.
 
 ### Native sandbox
 
-Native sandbox requires Wardby 0.5.4 or later.
+Native sandbox requires Wardby 0.6.0 or later.
 
 `up.sh` deploys everything the [native sandbox](native-sandbox.md) needs on
 this cluster. It builds and pushes the `native-worker` image, checks it (Node is
@@ -643,10 +600,10 @@ control plane is configured with `NATIVE_SANDBOX_LAUNCHER=kubernetes`,
 `NATIVE_SANDBOX_ENFORCEMENT_TIMEOUT_MS=60000`. Worker pods use the gVisor runtime
 class, `KUBERNETES_PLATFORM=gke-autopilot`, and the run priority class.
 
-- **One-time bootstrap.** The gateway has its own database login. After
-  `terraform apply` adds its identity to an existing deployment, run
-  `deploy/gke/bootstrap-database-iam.sh` once, then `deploy/gke/up.sh`; see
-  [Database login](#database-login).
+- **Database login.** The gateway has its own database login. `up.sh`
+  applies its `wardby_app` membership before the rollout, so adding its
+  identity to an existing deployment needs no manual step; see
+  [Upgrading an existing deployment](#upgrading-an-existing-deployment).
 - **Enable per agent.** Nothing runs in the sandbox until you opt an agent in:
   set its `nativeExecutionMode` to `sandbox` with the `update_agent` MCP tool
   (native agents only).
@@ -723,6 +680,13 @@ low-power mode until its rollout ends, and the mark stops it scaling them back
 down. With the mode on they stay up through the deploy (its database checks
 need them) and sleep one idle window later if nothing uses them.
 
+**Failed replicas.** Without the headroom placeholder, a rollout is more likely
+to hit Autopilot's admission race: the node rejects a new pod with `OutOfcpu`
+and the Deployment starts a replacement. The rejected pod stays behind as a
+Failed pod. After the rollouts, `up.sh` deletes Failed pods of the control
+plane, coding proxy and native gateway (never coding-run or sandbox pods), and
+the Viewer shows any such leftover as an old replica rather than a failure.
+
 **Sleeping now.** To skip the idle window (for example right after a test), an
 admin calls the MCP tool `power_sleep`, optionally with `target` set to `proxy`
 or `gateway` (default both). It scales the target to 0 at once, but never stops
@@ -792,6 +756,32 @@ that schedules onto a fresh node — one that has never pulled the image before
 cached. Raise `KUBERNETES_READY_TIMEOUT_MS` if you see runs fail
 `service_unready` only on new nodes.
 
+### Run dependency cache
+
+Off by default. To keep npm `node_modules` snapshots for coding runs
+(see [Dependency cache](coding-packages.md#dependency-cache)), set
+`dependency_cache_enabled = true` and a unique `dependency_cache_bucket_name` in
+`terraform.tfvars`, and deploy with `WARDBY_DEPENDENCY_CACHE=true
+deploy/gke/up.sh` (it grants the coding proxy's role the `DependencySnapshot`
+table before the rollout; see
+[Upgrading an existing deployment](#upgrading-an-existing-deployment)). Keep
+`dependency_cache_max_age_days` equal to the proxy's
+`WARDBY_DEPENDENCY_CACHE_MAX_AGE_DAYS` (default 14). Details:
+[`deploy/gke/README.md`](../deploy/gke/README.md#run-dependency-cache).
+
+### Session transcripts
+
+Off by default. To keep the transcripts of Claude Code agents with
+`sessionResume` on (see
+[Resume a run locally](coding-agent-setup.md#resume-a-run-locally)), set
+`session_transcripts_enabled = true` and a unique
+`session_transcripts_bucket_name` in `terraform.tfvars`, and deploy with
+`WARDBY_SESSION_TRANSCRIPTS=true deploy/gke/up.sh`. The bucket is separate from
+the dependency cache bucket and only the control plane can use it; its
+lifecycle rule and the control plane's `RUN_TRANSCRIPT_RETENTION_DAYS` both
+come from `session_transcripts_retention_days` (default 30). Details:
+[`deploy/gke/README.md`](../deploy/gke/README.md#session-transcripts).
+
 ### Roll back
 
 Images are pinned by digest, so undoing a rollout restores exactly what ran
@@ -803,11 +793,9 @@ kubectl -n wardby-coding rollout undo deploy/wardby-coding-proxy
 kubectl -n wardby-coding rollout undo deploy/wardby-native-gateway
 ```
 
-`rollout undo` restores only the Deployments' images and pod templates — there
-is no password path to fall back to any more, so a rollback is safe only for
-a version that still speaks IAM login. It does not touch any NetworkPolicy: a
-change to the database egress rules themselves is not undone by `rollout
-undo`.
+`rollout undo` restores only the Deployments' images and pod templates. It
+does not touch the database grants or any NetworkPolicy: a change to the
+database egress rules themselves is not undone by `rollout undo`.
 
 Database migrations only go forward: a rollback does not undo a schema change.
 That is safe while every migration is additive (new tables, nullable columns,