@wardby/cli 0.5.3 → 0.7.0

package/docs/getting-started-gke.md

@@ -3,6 +3,9 @@
 This is the supported Google Cloud deployment path for Wardby. It creates a
 GKE Autopilot cluster, Artifact Registry, private-IP Cloud SQL for PostgreSQL,
 isolated coding workers, an HTTPS Gateway, and the Wardby control plane.
+Before accepting production work, complete the
+[release verification](release-verification.md) and
+[security deployment](security-deployment.md) checks.
 
 The older `deploy/gcp` Cloud Run module is deprecated. New deployments should
 use `deploy/gke` and the `gke-autopilot` Kubernetes overlay described here.
@@ -14,7 +17,10 @@ use `deploy/gke` and the `gke-autopilot` Kubernetes overlay described here.
 ## Architecture
 
 - The Wardby control plane and coding proxy run in GKE Autopilot.
-- Each Codex coding run uses an ephemeral gVisor-backed pod.
+- Each Codex or Claude Code coding run uses an ephemeral gVisor-backed pod;
+  Claude Code adds a tool-runner sidecar in that pod.
+- Native agents set to sandbox mode run in single-use gVisor pods behind the
+  native gateway (see "Native sandbox" below).
 - Cloud SQL PostgreSQL has no public IP and is reached through private services
   access on the cluster's VPC.
 - Runtime, migration, and worker images are stored in Artifact Registry and
@@ -44,6 +50,7 @@ Install and authenticate:
 - `kubectl`
 - Helm 3 or later (installs External Secrets Operator)
 - Node.js 24 and npm
+- `curl`, `openssl` and `base64` (the database grants step uses them)
 
 You also need:
 
@@ -176,6 +183,41 @@ seeded: add it to the control plane's `env` in
 `deploy/kind-coding/manifests/overlays/gke-autopilot/control-plane.yaml` if you
 need it.
 
+To post [workflow notifications to Slack](slack-notifications.md), also add the
+Slack app's bot token. This requires Wardby 0.6.0 or later. It is optional:
+
+```dotenv
+WARDBY_SLACK_BOT_TOKEN="xoxb-..."
+```
+
+When it is in Secret Manager, `up.sh` syncs it into a separate Secret,
+`wardby-slack-env`, that only the control plane reads; when it is not, it
+leaves that Secret out and Slack stays off. `WARDBY_SLACK_CUSTOMIZE` and
+`WARDBY_SLACK_API_BASE_URL` are not secrets and are not seeded: add them to the
+control plane's `env` in the same `control-plane.yaml` if you need them. If you
+later remove the bot token, remove these two entries too: the control plane
+refuses to start with either one set and no token.
+
+`WARDBY_GITHUB_ALLOWED_ACCOUNTS` (optional, not a secret) adds GitHub accounts
+the App serves besides its owner. Pass it to `up.sh` on every run, for example
+`WARDBY_GITHUB_ALLOWED_ACCOUNTS=acme-labs WARDBY_HOSTNAME=app.example.com deploy/gke/up.sh`;
+a run without it serves only the App's owner. `up.sh` gives the same value to
+the control plane and to the native sandbox gateway, which runs sandboxed
+agents' repository tools. See
+[Which accounts the App serves](code-review-agents.md#which-accounts-the-app-serves-wardby_github_allowed_accounts).
+
+`WARDBY_MOBILE_ENABLED=true` (optional, not a secret) lets the Wardby mobile app
+sign in to the self-hosted OAuth server. Pass it to `up.sh` on every run, the
+same way as `WARDBY_GITHUB_ALLOWED_ACCOUNTS`; a run without it turns mobile
+sign-in off. It works for normal and low-power deploys. White-label builds that
+use their own redirect URI add `WARDBY_MOBILE_REDIRECT_URI` to the control
+plane's `env` themselves. See the help article `mobile-app`.
+
+`WARDBY_MENTION_EXPLANATIONS=false` (optional, not a secret) stops the GitHub
+App replying to say why a mention started nothing; pass it to `up.sh` on every
+run, the same way. See
+[What a linked review agent does](code-review-agents.md#what-a-linked-review-agent-does).
+
 Generate `SECRET_APP_KEY` with:
 
 ```sh
@@ -189,6 +231,13 @@ on. A value already in Secret Manager is never overwritten, so after the first
 deployment Secret Manager is the source of truth. You can then remove the
 production values from `.env.local`.
 
+External Secrets Operator runs as three small pods in the `external-secrets`
+namespace. `deploy/gke/eso-values.yaml` requests 100m CPU and 256 MiB of
+memory for each (memory limited to the same), because Autopilot bills a pod
+that requests nothing at 500m CPU and 2 GiB. If one is ever restarted for
+exceeding its memory limit, raise the limit and request together in that file
+and re-run `up.sh`; the Secrets it already synced are unaffected.
+
 Redeploying onto a cluster that already runs Wardby carries its existing
 `SECRET_APP_KEY` and login signing keys over, so stored credentials still
 decrypt and nobody has to sign in again.
@@ -249,9 +298,20 @@ The deployment script is idempotent. It converges Terraform, builds and pushes
 renders the GKE overlay, and waits for the proxy and control plane:
 
 ```sh
-HOSTNAME=wardby.example.com deploy/gke/up.sh
+WARDBY_HOSTNAME=wardby.example.com deploy/gke/up.sh
 ```
 
+Set `WARDBY_HOSTNAME`, not `HOSTNAME`: shells set `HOSTNAME` to the machine's own
+name, so the script no longer reads it. It refuses a hostname with no dot, or a
+local name such as `*.lan`, `*.local`, or `*.internal`. If you deployed with an
+earlier version using `HOSTNAME=...`, pass the same value as `WARDBY_HOSTNAME`.
+
+If the cluster is already published on a different hostname, the script stops
+before building anything: moving the hostname breaks DNS, every client's
+configured endpoint, and every issued token. In a terminal it asks you to type
+the new hostname; elsewhere, set `WARDBY_HOSTNAME_CHANGE` to the new hostname to
+confirm the move.
+
 Point the hostname's public A record at the reserved address:
 
 ```sh
@@ -263,10 +323,9 @@ Allow DNS and the managed certificate to converge before treating an HTTPS
 failure as an application failure.
 
 On a brand-new project, run `terraform apply` and then
-`bootstrap-database-iam.sh` before `up.sh`: the bootstrap has to grant the
-migrator before any migration can run. See "Database login" below for the
-exact order, including the database check `up.sh` doesn't pass until a second
-bootstrap run has granted the coding proxy too.
+`bootstrap-database-iam.sh` once before the first `up.sh`: the bootstrap has to
+grant the migrator before any migration can run. See "Database login" below
+for the exact order.
 
 ### Database login
 
@@ -275,19 +334,19 @@ through the Cloud SQL Auth Proxy, via Workload Identity from one Kubernetes
 service account. What each may do inside the database comes from a `NOLOGIN`
 group role in `deploy/gke/database-grants.sql`:
 
-| Workload      | Google service account   | Kubernetes service account | May do                                                                                                                                                                                          |
-| ------------- | ------------------------ | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| Control plane | `<name_prefix>-app`      | `wardby-control-plane`     | `wardby_app`: read/write every table, including the durable executor's in schema `dbos`; never change a schema                                                                                  |
-| Coding proxy  | `<name_prefix>-proxy`    | `wardby-coding-proxy`      | `wardby_proxy`: only its budget ledger — `CodingProxySession`/`CodingProxyRequest`/`RunModelUsage`, plus update `tokensIn`, `tokensOut`, `costUsd` and `turns` on `Run`, and read only its `id` |
-| Migrations    | `<name_prefix>-migrator` | `wardby-migrator`          | Acts as the table owner (`SET ROLE`), so `prisma migrate deploy` and `dbos schema` can alter and create tables                                                                                  |
+| Workload       | Google service account   | Kubernetes service account | May do                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
+| -------------- | ------------------------ | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Control plane  | `<name_prefix>-app`      | `wardby-control-plane`     | `wardby_app`: read/write every table, including the durable executor's in schema `dbos`; never change a schema                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
+| Coding proxy   | `<name_prefix>-proxy`    | `wardby-coding-proxy`      | `wardby_proxy`: its budget ledger (`CodingProxySession`/`CodingProxyRequest`/`RunModelUsage`: read, insert, update), the package registry's tables (`RegistryAllowance`, `RegistryFetch`, `RegistryAuditRecord`, `RegistryVersionFact`, `RegistryApprovedVersion`, `RegistryPlanRefusal`: read and insert only), the dependency cache's `DependencySnapshot` (read, insert, update, delete), update `tokensIn`, `tokensOut`, `costUsd` and `turns` on `Run` and read only its `id`, and read only the `runId`, `packageAllowlist`, `packagePolicy`, `repository`, `workerImage`, `toolImage`, `dependencyCache` and `packageAuditMode` columns of `CodingRun` |
+| Native gateway | `<name_prefix>-gateway`  | `wardby-native-gateway`    | Member of `wardby_app`: the same data access as the control plane, under its own login (see [Native sandbox](#native-sandbox))                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
+| Migrations     | `<name_prefix>-migrator` | `wardby-migrator`          | Acts as the table owner (`SET ROLE`), so `prisma migrate deploy` and `dbos schema` can alter and create tables                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
 
 `deploy/gke/bootstrap-database-iam.sh` applies the grants as the built-in
-owner, from a short-lived Job inside the cluster. Run it whenever
-`database-grants.sql` changes, before deploying the release that needs the new
-grants (`bootstrap-database-iam.sh --check`, then `bootstrap-database-iam.sh`,
-then `up.sh`), and as part of the orders below. A release deployed ahead of its
-grants fails with "permission denied" wherever it uses one it doesn't have yet.
-The grants run in one transaction, so a statement the database refuses leaves
+owner, from a short-lived Job inside the cluster. `up.sh` runs it on every
+deploy, after the migrations and before the rollout (see
+[Upgrading an existing deployment](#upgrading-an-existing-deployment)); you
+run it by hand only on a brand-new project, below, or with `--check`. The
+grants run in one transaction, so a statement the database refuses leaves
 nothing applied.
 
 The built-in owner is not a Terraform resource: `bootstrap-database-iam.sh`
@@ -299,110 +358,25 @@ ever stored, printed, or passed as a process argument.
 A brand-new project runs, in this order:
 
 1. `terraform -chdir=deploy/gke apply` — creates the cluster, the instance
-   and the three IAM database users. Run it yourself rather than through
+   and the four IAM database users. Run it yourself rather than through
    `up.sh`, which would go on to the migrations before the migrator has its
    grants.
-2. `deploy/gke/bootstrap-database-iam.sh` (default mode) — fetches the
-   cluster's kubectl credentials if you don't have them yet, creates the
-   owner, and applies the migrator's (and the app's) grants. The coding
-   proxy's ledger tables don't exist yet, so its grants are skipped: expected.
-3. `deploy/gke/up.sh` — applies the migrations and rolls out, then stops at
-   the database check: the coding proxy's grants are on tables that did not
-   exist in step 2.
-4. `deploy/gke/bootstrap-database-iam.sh` again — now that the tables exist,
-   also applies the coding proxy's ledger grants.
-5. `deploy/gke/up.sh` — rolls out again and passes every check.
-
-Two bootstrap runs, not one: a grant on a named table can't apply before the
-migration that creates that table has run.
-
-The same applies to every later release that adds a table the coding proxy
-uses: run `bootstrap-database-iam.sh` again after its migration. `up.sh`
-checks this for you. After the rollout it asks the database, as the proxy's
-own role, for every privilege `database-grants.sql` gives `wardby_proxy`
-(`deploy/gke/proxy-grant-checks.mjs`), and stops with the list of missing
-grants instead of letting the proxy fail registry requests with "permission
-denied".
-
-After upgrading an existing deployment to a release with cost attribution,
-re-run `bootstrap-database-iam.sh` so the coding proxy can record per-model
-usage for coding runs (used by the `cost_report` tool's `model` grouping).
-Until then coding runs still work, but their per-model breakdown isn't
-recorded.
+2. `deploy/gke/bootstrap-database-iam.sh` — fetches the cluster's kubectl
+   credentials if you don't have them yet, creates the owner, and applies the
+   migrator's (and the app's) grants. Grants on tables that don't exist yet
+   are skipped: expected.
+3. `deploy/gke/up.sh` — applies the migrations, then the grants again (now
+   that every table exists), then rolls out and passes every check.
 
 `bootstrap-database-iam.sh --check` runs the grants inside a transaction that
 is then rolled back, and reports success or the exact statement the database
-refused, without changing anything. Run it before the real run on a live
-deployment.
+refused, without changing any grant. Use it to see why a grants step failed.
 
 `connector_enforcement` (the Cloud SQL instance setting) defaults to
 `REQUIRED`: it refuses any connection that does not come through the Auth
 Proxy or a Cloud SQL connector, and with port 5432 to the instance closed in
 every NetworkPolicy, there's no route left for a plain Postgres connection to
-even attempt. Leave it at the default; an older deployment still on password
-login moves off it on an earlier revision of this module, below.
-
-### Moving an older deployment
-
-A deployment still on password login — from before this module retired it —
-can't adopt this revision directly: this revision no longer syncs the
-password `DATABASE_URL`, closes port 5432 and refuses non-Auth-Proxy
-connections, which would cut off pods still logging in with the password.
-It moves over in stages, because real pods are already serving traffic
-throughout.
-
-**1–2. Cut the running pods to IAM login, on the revision before this one.**
-Check out the revision of this module that added IAM login, before the
-password was retired: the merge of pull request #83 into `main`.
-`git log --oneline --first-parent main -- deploy/gke` lists the merges that
-changed this module, newest first; it's the one titled
-`Merge pull request #83 ...`, just below the merge that retired the password
-(`git log --oneline --merges --grep='#83' main` finds it directly). Then
-`git checkout <that commit>` and follow that revision's own
-`docs/getting-started-gke.md`, "Database login", and its order for "an
-existing deployment still on password login": `bootstrap-database-iam.sh
---password-from-stdin --check`, the same without `--check`, then `up.sh`.
-That rolls out password-less pods while the password still works as a
-fallback, and proves the control plane and the coding proxy each read the
-database through their own IAM login. That revision's
-`connector_enforcement` default is still `NOT_REQUIRED`, so nothing that
-still uses the password is cut off. Then return to this revision
-(`git checkout main`) for the last stage.
-
-**3. Retire the password**, once every pod speaks IAM login and this change
-has merged to `main`:
-
-1. Confirm nothing still uses the password: both Deployments' pods log in as
-   IAM users, and no pod has a password `DATABASE_URL` in its effective env.
-2. Delete the `database-url` Secret Manager secret:
-   `gcloud secrets delete <name_prefix>-database-url --quiet` — this module no
-   longer creates or reads it. Deleting it by hand first, before the next
-   step, is what lets Terraform drop it from state instead of trying to
-   destroy it.
-3. Remove any `connector_enforcement` line from `terraform.tfvars` (or set
-   it to `REQUIRED`), so the new default takes effect. Then `terraform plan`:
-   expect the generated password destroyed, the `database-url` secret and
-   its IAM binding gone from state (already deleted by hand), the old
-   password-login user forgotten (not destroyed), `connector_enforcement`
-   moving to `REQUIRED` on the instance (updated in place, not replaced),
-   and no other destroy. Review
-   the plan, then apply only once it matches that. If apply still fails
-   trying to destroy the `database-url` secret, the secret wasn't actually
-   deleted in step 2 — delete it and re-apply. Never turn off
-   `secrets_deletion_protection` to get past that error: it unprotects every
-   other secret in the same set, including `SECRET_APP_KEY` and the auth
-   keys, and losing `SECRET_APP_KEY` makes every credential already stored
-   in the database unreadable.
-4. `deploy/gke/up.sh` — new ExternalSecrets carry no `DATABASE_URL`,
-   NetworkPolicies no longer allow 5432, migrations, rollout, and the
-   database and endpoint checks.
-5. `deploy/gke/bootstrap-database-iam.sh` (default mode) — the guard now
-   passes, since the Secret no longer has a `DATABASE_URL`: it sets a
-   one-time owner password, reapplies the grants, and resets the password to
-   a value nobody holds.
-6. Verify: a direct password connection is refused (NetworkPolicy blocks
-   5432, and the instance refuses it too), the pods still log in through
-   IAM, and the Terraform state holds no database password.
+even attempt. Leave it at the default.
 
 ## 7. Verify
 
@@ -493,6 +467,38 @@ roll; if it fails, `up.sh` prints the `migrate` and `cloud-sql-proxy` container
 logs (and, if those are empty or the Job timed out, the Job's and pod's
 events), then stops.
 
+### Upgrading an existing deployment
+
+To upgrade, check out the new release and re-run `deploy/gke/up.sh`. No
+manual database step is needed: `up.sh` applies the migrations, then the
+database grants (`deploy/gke/bootstrap-database-iam.sh`, which applies
+`deploy/gke/database-grants.sql`), and only then rolls out the new pods. A
+release that adds a table, a column, or a workload that needs a new grant
+therefore never runs without it.
+
+- **If the migrations fail**, `up.sh` stops before the grants and the
+  rollout; the running version is untouched.
+- **If the grants fail**, `up.sh` stops before the rollout with a non-zero
+  exit; the running version keeps serving. The migrations already applied
+  are additive, so the running version tolerates them. Run
+  `deploy/gke/bootstrap-database-iam.sh --check` to see the exact statement
+  the database refused without changing anything, fix the cause, and re-run
+  `up.sh`.
+- **After the rollout**, `up.sh` checks that each new pod can read the
+  database, and asks the database, as the coding proxy's own role, for every
+  privilege `database-grants.sql` gives it
+  (`deploy/gke/proxy-grant-checks.mjs`). A missing grant stops the deploy
+  with a non-zero exit and the list of what's missing, rather than letting
+  the proxy fail registry requests with "permission denied".
+
+Never run two `up.sh` deploys against the same project at once: both would
+run migrations, set and reset the database owner's password, and roll out
+over each other. Nothing locks against it, so wait for one to finish.
+
+You run `bootstrap-database-iam.sh` by hand only on a brand-new project,
+between `terraform apply` and the first `up.sh` (see
+[Database login](#database-login)), or as `--check` for a dry run.
+
 ### Durable executor
 
 The control plane runs native runs on the durable executor (`EXECUTOR=dbos`):
@@ -517,24 +523,33 @@ the step that was in flight runs again. Coding runs are Kubernetes Jobs and are
   `up.sh` re-run whose source did not change, runs continue. When you deploy
   new code, the old pod first **drains**: after its 30-second `preStop` it
   stops the scheduler and waits for the runs it is executing to finish, up to
-  `SHUTDOWN_DRAIN_SECONDS` (default 600), while the new pod serves traffic.
-  An idle pod exits at once. A run still going after that limit is marked
-  `lost`; if an `@`-mention started it, its status comment on GitHub says the
-  request was interrupted and should be repeated. The manifest's
-  `terminationGracePeriodSeconds` (720) must stay above the drain limit plus
-  the `preStop` delay.
+  `SHUTDOWN_DRAIN_SECONDS` (540 in the GKE manifest), while the new pod
+  serves traffic. Open HTTP connections, such as a connected MCP client's
+  stream, get 10 seconds to finish and are then closed; clients reconnect to
+  the new pod.
+  An idle pod exits at once. A native run that is waiting in a `delegate_to_*`
+  call on a coding sub-agent or a sandbox-mode native sub-agent is taken over
+  by the new version: it continues when the sub-agent finishes, without
+  repeating the turns it already took. A run that was mid-turn, waiting on
+  another tool, or waiting on a control-plane native sub-agent is marked
+  `lost` rather than taken over, as is a taken-over run whose recorded steps
+  no longer match the new code, and a run still going after the drain limit.
+  The sub-agent runs a lost run started are stopped (reason
+  `parent run lost`). If an `@`-mention started a lost run, its status comment
+  on GitHub says the request was interrupted and should be repeated. The
+  manifest's `terminationGracePeriodSeconds` (600) must stay above the drain
+  limit plus the `preStop` delay. Autopilot does not allow more than 600: a
+  larger value is lowered to 600, so to change the budget, lower
+  `SHUTDOWN_DRAIN_SECONDS` rather than raising the grace period.
 - **Data at rest and retention.** `dbos.operation_outputs` holds every step's
   output — prompts, model responses and full tool results — with no retention
   limit. Pruning finished workflows is the operator's job; see
   [Durable executor](security-deployment.md#durable-executor) for what is
   stored and how to prune it with `DBOS.deleteWorkflow`.
 
-**Enabling it on an existing deployment.** The grants changed, so apply them
-before deploying: run `deploy/gke/bootstrap-database-iam.sh --check`, then
-`deploy/gke/bootstrap-database-iam.sh`, then `deploy/gke/up.sh`. Without the
-grants the new control-plane pod crash-loops with "permission denied" while
-the old one keeps serving, and `up.sh` stops at the rollout. On a brand-new
-project the order under "Database login" already covers it.
+**Enabling it on an existing deployment.** Re-run `deploy/gke/up.sh`: it
+applies the durable executor's grants before the rollout (see
+[Upgrading an existing deployment](#upgrading-an-existing-deployment)).
 
 **Switching back** to the in-process executor: set `EXECUTOR` to `in-process`
 in `deploy/kind-coding/manifests/overlays/gke-autopilot/control-plane.yaml`
@@ -568,6 +583,167 @@ it, set `replicas: 0` in
 A PriorityClass's value and preemption policy cannot be changed in place:
 delete the class and re-run `up.sh` to change them.
 
+### Native sandbox
+
+Native sandbox requires Wardby 0.5.4 or later.
+
+`up.sh` deploys everything the [native sandbox](native-sandbox.md) needs on
+this cluster. It builds and pushes the `native-worker` image, checks it (Node is
+present; `curl`, `wget`, `nc` and `ssh` are not), and pins it by digest as
+`NATIVE_SANDBOX_WORKER_IMAGE`. It syncs the gateway's ExternalSecret, rolls out
+`wardby-native-gateway`, waits for it, and checks its database round trip. The
+control plane is configured with `NATIVE_SANDBOX_LAUNCHER=kubernetes`,
+`NATIVE_SANDBOX_CPUS=0.5`, `NATIVE_SANDBOX_MEMORY_MB=512`,
+`NATIVE_SANDBOX_MAX_CONCURRENT=4`, `NATIVE_SANDBOX_READY_TIMEOUT_MS=600000`
+(a cold gVisor node takes about two minutes), and
+`NATIVE_SANDBOX_ENFORCEMENT_TIMEOUT_MS=60000`. Worker pods use the gVisor runtime
+class, `KUBERNETES_PLATFORM=gke-autopilot`, and the run priority class.
+
+- **Database login.** The gateway has its own database login. `up.sh`
+  applies its `wardby_app` membership before the rollout, so adding its
+  identity to an existing deployment needs no manual step; see
+  [Upgrading an existing deployment](#upgrading-an-existing-deployment).
+- **Enable per agent.** Nothing runs in the sandbox until you opt an agent in:
+  set its `nativeExecutionMode` to `sandbox` with the `update_agent` MCP tool
+  (native agents only).
+- **Capacity.** At most `NATIVE_SANDBOX_MAX_CONCURRENT` sandbox runs are active
+  at once; a start past that fails with `native_sandbox_capacity` rather than
+  queuing. The `wardby-coding` ResourceQuota (26 pods, 11 CPU, 20Gi memory)
+  already covers the gateway's two replicas and that many 500m / 512Mi worker
+  pods. Raise the quota together with `NATIVE_SANDBOX_MAX_CONCURRENT` or
+  `CODING_MAX_CONCURRENT`.
+- **Warm pool.** To keep idle, already-isolated workers ready so runs skip pod
+  scheduling and the isolation probe, set `NATIVE_SANDBOX_WARM_POOL_SIZE` (0 to
+  50, default 0) in the control-plane manifest and raise the `wardby-coding`
+  ResourceQuota by that many 500m / 512Mi pods. Each idle worker is billed
+  continuously. See [Warm pool](native-sandbox.md#warm-pool).
+- **Check isolation.** `npm run test:native-cluster` runs launcher-only
+  isolation checks against the deployed gateway; set `NATIVE_TEST_WORKER_IMAGE`
+  to the worker digest and `NATIVE_TEST_FORBIDDEN` to the database's private
+  `host:port` (for example `10.0.0.5:3307`).
+- **Roll back.** Set an agent back to `control-plane` with `update_agent`; this
+  affects only new runs, and runs already started in sandbox mode finish there.
+  To undo a bad gateway release, run
+  `kubectl -n wardby-coding rollout undo deploy/wardby-native-gateway`.
+
+### Low-power mode
+
+Low-power mode requires Wardby 0.6.0 or later.
+
+Low-power mode cuts the idle cost of a deployment that runs agents only now and
+then. The coding proxy and the native gateway scale to zero replicas when
+nothing needs them, and wake on demand; the headroom placeholder is turned off;
+and the control plane runs 1 copy, scaling to 2 under load. The database and
+edge are unchanged.
+
+Turn it on by deploying with the flag:
+
+```sh
+WARDBY_HOSTNAME=wardby.example.com WARDBY_LOW_POWER=1 deploy/gke/up.sh
+```
+
+To turn it off, re-run `up.sh` without the flag. That removes the autoscaler
+(HorizontalPodAutoscaler `wardby-control-plane`) and the Role and RoleBinding
+`wardby-power`, and restores the normal sizes: proxy 1, gateway 2, headroom 1,
+control plane 1.
+
+| Variable                 | Where it is set                    | Values                          | Meaning                                                                                                                                                  |
+| ------------------------ | ---------------------------------- | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `WARDBY_LOW_POWER`       | `up.sh` environment; control plane | unset, `0` (off) or `1` (on)    | Anything else fails `up.sh` and stops the control plane at start-up. Needs `JOB_LAUNCHER=kubernetes`, else it logs `low_power_ignored` and does nothing. |
+| `LOW_POWER_IDLE_MINUTES` | Control-plane Deployment           | integer 5 to 1440, default `30` | How long a target must be unused before it sleeps. Read only when `WARDBY_LOW_POWER=1`.                                                                  |
+
+**Sleeping.** Every minute the scheduler leader scales the coding proxy
+(`wardby-coding-proxy`) to 0 when no coding run is pending or running and none
+finished within the idle window. It scales the native gateway
+(`wardby-native-gateway`) to 0 when no sandbox native run is pending, running or
+recently finished and the gateway has no active session. Either target sleeps
+only once at least one idle window has passed since it was last woken (recorded
+in its `wardby.io/power-woke-at` annotation) and since the control plane
+started, whichever is later. The condition is checked again immediately before
+scaling. The control plane
+logs `power: proxy slept (idle 30m)`.
+
+**Waking.** Starting a coding run scales the proxy to 1 and waits for a ready
+endpoint. Starting a sandbox run does the same for the gateway. The wait is at
+least 5 minutes, because a cold Autopilot node can add 1 to 2 minutes of
+start-up; a `KUBERNETES_READY_TIMEOUT_MS` (proxy) or
+`NATIVE_SANDBOX_READY_TIMEOUT_MS` (gateway) above 5 minutes, set on the
+control-plane Deployment, raises it to that value. The first coding run after a
+control-plane restart also runs the cluster preflight canary, because start-up
+preflight is deferred while the proxy sleeps (`kubernetes_preflight_deferred`).
+
+**Deploys.** Every `up.sh` run marks the proxy and gateway as just woken before
+it applies the manifests, which bring them back up. That includes a run that
+turns low-power mode off: the control plane being replaced still runs in
+low-power mode until its rollout ends, and the mark stops it scaling them back
+down. With the mode on they stay up through the deploy (its database checks
+need them) and sleep one idle window later if nothing uses them.
+
+**Failed replicas.** Without the headroom placeholder, a rollout is more likely
+to hit Autopilot's admission race: the node rejects a new pod with `OutOfcpu`
+and the Deployment starts a replacement. The rejected pod stays behind as a
+Failed pod. After the rollouts, `up.sh` deletes Failed pods of the control
+plane, coding proxy and native gateway (never coding-run or sandbox pods), and
+the Viewer shows any such leftover as an old replica rather than a failure.
+
+**Sleeping now.** To skip the idle window (for example right after a test), an
+admin calls the MCP tool `power_sleep`, optionally with `target` set to `proxy`
+or `gateway` (default both). It scales the target to 0 at once, but never stops
+a run: a target with a pending or running run, or for the gateway an active
+gateway session, is left up and its run ids are returned under `blocked`.
+Recently finished runs do not block it. The next run wakes the target as usual.
+`power_status` shows each target's replicas, readiness, last wake time and
+blocking run ids. Both tools need the `power:manage` scope, which only the
+`admin` role grants; with the mode off, `power_sleep` fails with
+`409 low_power_disabled` and `power_status` returns `{ "enabled": false }`.
+Each `power_sleep` is logged as `event: "power.sleep"` with the caller's
+principal.
+
+**Control-plane autoscaling.** The control plane scales between 1 and 2 copies
+on CPU (20% target, scale-down after 10 minutes). A run executes on
+the copy that started it, so scheduled runs stay on the scheduler leader, while
+runs started by a webhook or over MCP spread across copies. While the mode is on
+the autoscaler owns the replica count, so do not set it by hand.
+
+**Trade-offs.**
+
+- Headroom is off, so the control plane has less protection against being
+  preempted when the cluster is short of capacity (see
+  [Pod priority and headroom](#pod-priority-and-headroom)).
+- The gateway runs 1 replica instead of 2 while awake, so there is no
+  redundancy.
+- `NATIVE_SANDBOX_WARM_POOL_SIZE` must be 0; the control plane refuses to start
+  otherwise, because a warm pool needs the gateway running.
+- The gateway must be in the same namespace as `KUBERNETES_NAMESPACE`.
+- The control plane needs the `wardby-power` Role: `get` and `patch` on the
+  proxy and gateway Deployments, and `get` on their Endpoints (to tell when a
+  woken target is ready). If any of it is missing, start-up warns
+  `low_power_scale_forbidden: missing ...` naming each permission, and waking,
+  `power_status` and `power_sleep` fail with that error; re-run `up.sh` with
+  the flag.
+
+**Cost.** Autopilot bills Pod requests while a Pod exists. The table uses
+us-central1 list prices of $0.0445 per vCPU-hour and $0.0049225 per GiB-hour and
+730 hours a month; substitute your region's current rates. Requests include the
+Cloud SQL sidecar.
+
+| Workload                     | Requests per replica  | Per hour | Per month, always on | In low-power mode                       |
+| ---------------------------- | --------------------- | -------- | -------------------- | --------------------------------------- |
+| Control plane                | 1.1 vCPU / 2.125 GiB  | $0.0594  | $43.37               | 1 copy always; $0.0594/h per extra copy |
+| Coding proxy                 | 1.1 vCPU / 3.125 GiB  | $0.0643  | $46.96               | $0.0643/h only while awake              |
+| Native gateway (per replica) | 0.35 vCPU / 0.625 GiB | $0.0187  | $13.62 (x2 = $27.23) | 1 replica, $0.0187/h only while awake   |
+| Headroom placeholder         | 0.25 vCPU / 0.5 GiB   | $0.0136  | $9.92                | $0                                      |
+
+With the mode off, those four workloads cost about $127.48 a month at these
+rates. In low-power mode the control plane is still always on, and a fully
+awake moment costs about $0.142 an hour for one control-plane copy, the proxy
+and one gateway replica. Your total depends on how many hours the proxy and
+gateway stay awake. Cloud SQL, the static address, load balancer, Cloud Armor
+and any other always-on component are unchanged.
+
+For error codes and log lines, see the
+[low-power mode help article](../help/low-power-mode.md).
+
 ### Coding-run services
 
 A coding run whose agent allows [services](coding-services.md) waits for each
@@ -579,6 +755,32 @@ that schedules onto a fresh node — one that has never pulled the image before
 cached. Raise `KUBERNETES_READY_TIMEOUT_MS` if you see runs fail
 `service_unready` only on new nodes.
 
+### Run dependency cache
+
+Off by default. To keep npm `node_modules` snapshots for coding runs
+(see [Dependency cache](coding-packages.md#dependency-cache)), set
+`dependency_cache_enabled = true` and a unique `dependency_cache_bucket_name` in
+`terraform.tfvars`, and deploy with `WARDBY_DEPENDENCY_CACHE=true
+deploy/gke/up.sh` (it grants the coding proxy's role the `DependencySnapshot`
+table before the rollout; see
+[Upgrading an existing deployment](#upgrading-an-existing-deployment)). Keep
+`dependency_cache_max_age_days` equal to the proxy's
+`WARDBY_DEPENDENCY_CACHE_MAX_AGE_DAYS` (default 14). Details:
+[`deploy/gke/README.md`](../deploy/gke/README.md#run-dependency-cache).
+
+### Session transcripts
+
+Off by default. To keep the transcripts of Claude Code agents with
+`sessionResume` on (see
+[Resume a run locally](coding-agent-setup.md#resume-a-run-locally)), set
+`session_transcripts_enabled = true` and a unique
+`session_transcripts_bucket_name` in `terraform.tfvars`, and deploy with
+`WARDBY_SESSION_TRANSCRIPTS=true deploy/gke/up.sh`. The bucket is separate from
+the dependency cache bucket and only the control plane can use it; its
+lifecycle rule and the control plane's `RUN_TRANSCRIPT_RETENTION_DAYS` both
+come from `session_transcripts_retention_days` (default 30). Details:
+[`deploy/gke/README.md`](../deploy/gke/README.md#session-transcripts).
+
 ### Roll back
 
 Images are pinned by digest, so undoing a rollout restores exactly what ran
@@ -587,13 +789,12 @@ before. `up.sh` prints the previous digests at the end of every deploy.
 ```sh
 kubectl -n wardby-coding rollout undo deploy/wardby-control-plane
 kubectl -n wardby-coding rollout undo deploy/wardby-coding-proxy
+kubectl -n wardby-coding rollout undo deploy/wardby-native-gateway
 ```
 
-`rollout undo` restores only the Deployments' images and pod templates — there
-is no password path to fall back to any more, so a rollback is safe only for
-a version that still speaks IAM login. It does not touch any NetworkPolicy: a
-change to the database egress rules themselves is not undone by `rollout
-undo`.
+`rollout undo` restores only the Deployments' images and pod templates. It
+does not touch the database grants or any NetworkPolicy: a change to the
+database egress rules themselves is not undone by `rollout undo`.
 
 Database migrations only go forward: a rollback does not undo a schema change.
 That is safe while every migration is additive (new tables, nullable columns,
@@ -623,7 +824,9 @@ kubectl -n wardby-coding rollout restart deploy/wardby-coding-proxy deploy/wardb
 
 For a Jira setting, add the version to its `wardby-jira-*` secret, wait on
 `wardby-jira-env` instead, and restart only `deploy/wardby-control-plane`.
-Rotate the token and its `jira-api-token-expires-at` together.
+Rotate the token and its `jira-api-token-expires-at` together. The Slack bot
+token works the same way: add the version to `wardby-slack-bot-token`, wait on
+`wardby-slack-env`, and restart only `deploy/wardby-control-plane`.
 
 Pods read their environment only at start, hence the restart. The LLM API
 keys and the database URL are read by both Deployments; the other secrets only