@wardby/cli 0.6.0 → 0.8.0

package/docs/mcp-servers.md

@@ -0,0 +1,254 @@
+# Remote MCP servers for native and coding agents
+
+A native, Claude Code or Codex agent can call the tools of a remote MCP server. You register the
+server once (its HTTPS URL plus your brokered secrets), Wardby saves the
+server's tool list, and you attach an allowlisted subset of those tools to
+an agent. A native agent sees each tool as `<server name>__<tool name>`; coding agents see
+them as described in [Coding agents](#coding-agents).
+
+`WARDBY_MCP_SERVERS` decides who may register servers: `owners` (the
+default, anyone), `admins` (registering, updating and refreshing need
+`hosts:admin`) or `off` (every tool below is refused except
+`detach_mcp_server` and `delete_mcp_server`, so cleanup stays possible, and
+attached servers' tools are left out of runs).
+
+The tools below need the `tools:write` scope to change servers and
+attachments and `agents:read` to list and get them.
+
+## Tools
+
+| Tool                  | Who                   | What it does                                                                                                       |
+| --------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------ |
+| `register_mcp_server` | anyone (see mode)     | Registers a server from `name`, `url` (e.g. `https://mcp.example.com/mcp`) and `secretNames`; saves its tool list. |
+| `update_mcp_server`   | the server's owner    | Changes the URL or secret names, saves a fresh tool list and returns what changed.                                 |
+| `refresh_mcp_server`  | the server's owner    | Saves a fresh tool list straight away; returns added, removed and changed tools and the agents left missing tools. |
+| `list_mcp_servers`    | anyone                | Your servers and those shared with you: id, name, host, tool count, saved-at, `yours`.                             |
+| `get_mcp_server`      | owner, or `read`+     | One server with its full saved tool list (secret names and URL only for the owner).                                |
+| `delete_mcp_server`   | the server's owner    | Deletes a server and detaches it from every agent. Works when the mode is `off`.                                   |
+| `attach_mcp_server`   | agent's owner; `use`+ | Lets a native or coding agent call the named tools (`allowedTools`, exact names) with an optional `callTimeoutMs`. |
+| `detach_mcp_server`   | the agent's owner     | Removes a server's tools from an agent. Works when the mode is `off`.                                              |
+
+`get_agent` shows each attached server under `mcpServers`: the server's `serverId`
+and name, the allowed tools, `callTimeoutMs`, the exposed tool names and any allowed tools the server no
+longer lists (`missingTools`). Anyone with read access on the agent sees this.
+It never includes URLs, secret names or values, or saved tool lists.
+
+## Create a secret for the server
+
+Servers are authenticated with [brokered secrets](tool-secrets.md): Wardby adds
+the value to every request to the server's host and nobody, including the
+model, can read it back. Create the secret, then set its broker config with
+`set_secret_broker` so it is limited to the server's host. For a bearer token:
+
+```json
+{
+  "hosts": ["mcp.example.com"],
+  "placement": { "kind": "header", "name": "Authorization", "format": "Bearer {value}" }
+}
+```
+
+A server that wants two headers needs two secrets, each with its own
+placement (for example `X-Api-Key` with `format: "{value}"`, and
+`X-Workspace` the same way); name both in `secretNames` (at most 8 names, each up to 1024 characters). The URL itself can't
+carry credentials: a username, password or fragment is refused.
+
+## Register, read the tool list, attach
+
+1. `register_mcp_server` with `name` (1-32 characters: lower-case letters,
+   digits, `_` or `-`), `url` (`https://mcp.example.com/mcp`: HTTPS only,
+   default port) and `secretNames`. Wardby calls the server's `tools/list`
+   and saves the result. A secret that is missing, not brokered, or not
+   brokered to the URL's host is refused.
+2. `get_mcp_server` shows the saved tools: name, description, input schema and
+   the server's annotations.
+3. `attach_mcp_server` with the agent, the server and `allowedTools`: exact
+   tool names from the saved list, at least one, no wildcards. Attaching
+   replaces the agent's earlier allowlist and time limit for that server.
+   Native, Claude Code and Codex agents can attach a server (see [Coding agents](#coding-agents)).
+
+The agent sees each tool as `<server name>__<tool name>`, with any character
+outside letters, digits, `_` and `-` changed to `_`; the result must be at
+most 64 characters. The description starts with `From MCP server <server name>: `.
+Attaching is refused if the name clashes with another tool on the agent
+(attached, another MCP tool, or a built-in name).
+
+## Coding agents
+
+A Claude Code or Codex agent can use attached servers too. Attach them the same way, with
+`attach_mcp_server`. The tools reach the run like this:
+
+1. The worker calls the coding proxy's `/mcp` route with a per-run MCP token (`rmt_…`, in the
+   worker's `WARDBY_MCP_TOKEN`). Workers still reach only the proxy.
+2. The proxy forwards the request to the native gateway's `/coding-mcp` route. The proxy never sees
+   a server's URL or secrets. The gateway checks the token again, checks the live attachment and
+   runs the call, and removes secret values from the result before it leaves.
+
+The token belongs to one run: it works only on `/mcp`, only while the run is active and before its
+deadline, and only its hash is stored.
+
+- **Claude Code agents** see each tool as `mcp__wardby_mcp__<server name>__<tool name>` and may use
+  any allowed tool. The 64-character limit on `<server name>__<tool name>` is the same as for
+  native agents; the prefix adds no other limit.
+- **Codex agents may use only tools the server marks `readOnlyHint: true`.** Wardby enforces this
+  when you attach (an allowlist with another tool is refused), when a run loads its tools (a tool
+  whose saved annotations no longer say read-only, for example after `refresh_mcp_server`, is left
+  out) and on every call. The reason is that repository code running in a Codex run can read the
+  run's MCP token, so only tools that don't change anything are safe to give it. `readOnlyHint` is
+  the server's own claim, which Wardby can't verify. The allowlist stays your control: allow only
+  tools you trust to be read-only.
+
+  A run follows the provider it was started with, even if the agent's provider changes before the
+  run launches. A coding agent with no coding profile is treated as Codex when you attach.
+
+- **Switching an agent to Codex.** When `update_agent` makes an agent a Codex agent (changing
+  `codingProfile.provider` to `codex`, or making a native agent a coding agent, which uses Codex by
+  default) and some of its attached tools aren't marked read-only, the change still goes ahead and
+  the result's `warnings` names those tools: "These MCP tools aren't read-only and won't be
+  available to Codex runs: …". They stay attached but Codex runs don't get them. Detach them, or
+  re-attach the server with only read-only tools, to remove the warning's cause.
+- **Call limit.** A run may make `WARDBY_MCP_MAX_CALLS_PER_RUN` calls (default `100`). Every call
+  counts, whatever its outcome. Past the limit a call returns
+  [`mcp_call_limit`](../help/errors/mcp-call-limit.md); the run goes on.
+- **Proxy setting.** The coding proxy needs `WARDBY_MCP_GATEWAY_URL`, the native gateway's base URL
+  (the kind and GKE manifests set it). Without it `/mcp` answers `503` with
+  [`mcp_unavailable`](../help/errors/mcp-unavailable.md) and coding agents get no MCP tools. See
+  [Remote MCP servers](security-deployment.md#remote-mcp-servers) for the variables.
+- **Worker images.** MCP for coding agents needs this release's Claude worker image and a new Codex
+  driver image. An older image ignores the token and the agent gets no MCP tools; the run does not
+  fail. Update the images together with the server.
+- **Repository configuration.** A repository's own MCP configuration (such as `.mcp.json`) is still
+  never loaded; the attached servers are the only MCP tools a coding run has.
+- **Codex models that use tool search** load MCP tools on demand: Codex sends a client-side
+  `tool_search` tool and adds the matching MCP tools to the conversation when the model searches.
+  This needs this release's coding proxy; an older proxy refuses those models' requests with
+  `openai_tool_not_allowed:tool_search`, MCP or not.
+
+Time limits and the other rules in this guide apply to coding agents too. See
+[Coding worker isolation](coding-worker-isolation.md#remote-mcp-tools) for the security notes.
+
+## Time limits
+
+Each call has `callTimeoutMs` (default `30000`, set on `attach_mcp_server`).
+It must be at least `1000` and can't exceed `WARDBY_TOOL_MAX_TIMEOUT_MS`; a call runs with the lower of
+the two, and never past the run's own deadline. `stop_run` aborts a call in
+flight. A call that runs out of time returns `mcp_call_timeout` to the model
+and the run goes on.
+
+A coding agent's calls have one more limit: the coding proxy gives up on a forwarded call after 5
+minutes, whatever `WARDBY_TOOL_MAX_TIMEOUT_MS` and `callTimeoutMs` say. The call then fails with
+[`mcp_unavailable`](../help/errors/mcp-unavailable.md) (`502`), the gateway stops the call, and it
+still counts against the run's call limit. A run that has been stopped or cancelled can't start new
+calls: its token is refused with `401`.
+
+## Refreshing and changes
+
+`refresh_mcp_server` (and `update_mcp_server`) fetch the tool list again and
+apply it at once. The reply lists tools added, removed and changed
+(description or schema). A removed tool that an agent allows is dropped from
+that agent's runs and shows under `missingTools` in `get_agent`; nothing
+needs re-attaching if the server brings it back and you refresh again. A
+tool the server newly adds is not available to any agent until you attach it.
+
+## What a call does
+
+Each call opens its own session: `initialize`, `notifications/initialized`,
+`tools/call`, then a `DELETE` to end it. Because every call gets a fresh
+session, a server that keeps state between calls in one session won't see
+that state on the next call. Wardby checks on every call that the tool is
+still attached and still in the saved list, and that the agent's owner still
+owns the server or holds `use` on it; the call uses the server's current URL
+and secrets. Arguments that aren't a JSON object return `invalid_arguments`
+without calling the server. Text returned to the model is cut at 100,000
+characters, with a note.
+
+Wardby answers none of a server's own requests (sampling, elicitation,
+roots). Notifications in a reply are ignored. A reply that carries only a
+request from the server, and no answer to Wardby's request, fails the call
+with `mcp_server_error`.
+
+One log line per call records the run id, agent id, server id, tool,
+outcome, duration and result size, never the arguments, the response or the
+URL. Secret values are removed from results, errors and logs.
+
+## Not supported
+
+stdio servers, OAuth to a server (use a header secret), MCP resources,
+prompts and sampling.
+
+## Trust
+
+A server's results are untrusted input to the model, like any tool result. The
+allowlist is the control: attach only the tools an agent needs. A server's
+annotations (read-only, destructive and so on) are hints from the server, not
+something Wardby can verify. The one place Wardby acts on a hint is Codex coding
+agents, which may use only tools marked `readOnlyHint: true` (see
+[Coding agents](#coding-agents)).
+
+## Sharing a server
+
+Share a server you own with `grant_access` (`resourceType: "mcp_server"`, a
+principal or `everyone`, level `read`, `use` or `write`; `everyone` is capped
+at `use`). `read` lets the grantee see the server and its tool list; `use`
+also lets them attach it to their own agents. The server's tools then run with
+your brokered secrets, so grant `use` only to people you trust with them.
+`write` allows nothing beyond `use`: only the owner updates, refreshes,
+deletes or re-grants a server.
+
+Sharing uses `grant_access` and `revoke_access`, which need the
+`agents:write` scope; registering and attaching need `tools:write`.
+
+An agent may use a server only while its owner owns the server or holds `use`
+(or `write`) on it. `revoke_access`, and `grant_access` lowering a grant to
+`read` (for a principal, or `everyone` from `use` to `read`), detach the
+server at once from every agent whose owner can no longer use it; the reply
+counts them under `attachmentsRemoved`. Your own agents, and agents whose
+owner still holds `use` through another grant, keep it. Runs already going
+lose the tools on their next call, which returns `mcp_tool_not_allowed`.
+
+`delete_mcp_server` removes the server's tools from every agent attached to
+it, including other owners' agents that use it through a grant, and removes
+its grants.
+
+When the server's secrets can't be loaded (a secret was deleted, or its
+broker config no longer lists the server's host), the owner's own agents get
+an `mcp_server_error` that names the secret. An agent using the server through
+a grant gets only "The MCP server's credentials can't be used; ask its owner
+to check it.", never the owner's secret names.
+
+When a secret exists but the Wardby process making the call can't decrypt its
+stored value, the call returns `mcp_server_credentials_unavailable`, and
+`register_mcp_server`, `update_mcp_server` and `refresh_mcp_server` fail with
+it as an HTTP 500 error. Every process that decrypts secrets (control plane,
+scheduler, native gateway) must use the same `SECRET_APP_KEY`; if the key
+changed, set the secret again. The owner's agents get that advice; an agent
+using the server through a grant gets the generic message above. Each failure
+also writes one operator `warn` log line with
+`reason: "mcp_secret_decrypt_failed"`, the server and the secret name.
+
+An attachment acts with the server owner's secrets. When `make_owner` gives an
+agent a new owner, servers the new owner neither owns nor holds `use` on are
+detached and listed under `bindingsRemoved` with `kind: "mcp_server"`.
+`wardby grants adopt-public` and `wardby grants prune-bindings` remove them the
+same way. An agent that holds an MCP server can't be made writable by
+everyone.
+
+## Troubleshooting
+
+A failed call returns `{"error": "<code>", "message": "..."}` to the model;
+the run goes on.
+
+| Code                                                                                         | Meaning                                                               |
+| -------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
+| [`mcp_servers_off`](../help/errors/mcp-servers-off.md)                                       | `WARDBY_MCP_SERVERS=off`.                                             |
+| [`mcp_server_unreachable`](../help/errors/mcp-server-unreachable.md)                         | Wardby couldn't connect to the server.                                |
+| [`mcp_server_auth_failed`](../help/errors/mcp-server-auth-failed.md)                         | The server answered 401 or 403.                                       |
+| [`mcp_server_credentials_unavailable`](../help/errors/mcp-server-credentials-unavailable.md) | A secret can't be decrypted: check every process's `SECRET_APP_KEY`.  |
+| [`mcp_server_error`](../help/errors/mcp-server-error.md)                                     | Another HTTP error, a malformed reply, or the tool reported an error. |
+| [`mcp_tool_not_allowed`](../help/errors/mcp-tool-not-allowed.md)                             | Detached, off the allowlist, or the owner lost `use` on the server.   |
+| [`mcp_tool_removed`](../help/errors/mcp-tool-removed.md)                                     | The server no longer lists the tool.                                  |
+| [`mcp_call_timeout`](../help/errors/mcp-call-timeout.md)                                     | The call hit its time limit or the run was stopped.                   |
+| [`mcp_call_limit`](../help/errors/mcp-call-limit.md)                                         | A coding run made more calls than `WARDBY_MCP_MAX_CALLS_PER_RUN`.     |
+| [`mcp_unavailable`](../help/errors/mcp-unavailable.md)                                       | The coding proxy has no gateway to forward MCP calls to.              |
+
+See also [Secrets and tools](tool-secrets.md) and
+[Remote MCP servers](security-deployment.md#remote-mcp-servers).

package/docs/native-sandbox.md

@@ -1,6 +1,6 @@
 # Native sandbox
 
-> **Requires Wardby 0.5.4 or later.**
+> **Requires Wardby 0.6.0 or later.**
 
 By default a native agent runs inside the Wardby server process: its turn loop
 and any user tool code you attach execute next to the database connection, the
@@ -78,6 +78,13 @@ container to it under the alias `wardby-native-gateway`, and starts the worker
 on it. When the run ends the worker, the network, and the gateway's membership
 of it are removed.
 
+The gateway also serves coding agents' remote MCP tools: the coding proxy
+forwards a coding run's `/mcp` requests to the gateway's HTTP port, on the
+`/coding-mcp` route, where the run's MCP token is checked again. Set
+`WARDBY_MCP_GATEWAY_URL` on the coding proxy to the gateway's base URL
+([details](security-deployment.md#remote-mcp-servers)). The proxy holds no
+gateway credentials, and each route authenticates separately.
+
 The gateway is stateless: all state is in Postgres. You can restart it while
 runs are in flight; workers retry their calls. It must not be given the Docker
 socket, because workers can reach it.
@@ -131,17 +138,15 @@ The gateway is the same Wardby build as the server, started with the
 `native-gateway` command. It needs the server's database, LLM, secrets, and
 integration settings, and **no Docker access**.
 
-| Gateway variable        | Purpose                                                                                                                                                                                                                                                    |
-| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `NATIVE_GATEWAY_LISTEN` | `host:port` to listen on. Default `0.0.0.0:8790`.                                                                                                                                                                                                          |
-| `DATABASE_URL`          | The same database the server uses. With `DATABASE_AUTH=rds-iam` the gateway also needs `DATABASE_AUTH`, `AWS_REGION` (or `DATABASE_AWS_REGION`) and the CA bundle file named by `sslrootcert` ([details](security-deployment.md#database-authentication)). |
-| LLM provider keys       | The same provider settings as the server (for example `OPENAI_API_KEY`).                                                                                                                                                                                   |
-| `SECRET_APP_KEY`        | Same value as the server, so tools can read their bound secrets or send brokered ones ([details](tool-secrets.md)).                                                                                                                                        |
-| Integration settings    | The same repository-host and issue-tracker settings as the server.                                                                                                                                                                                         |
-
-If sandboxed agents delegate to coding agents, give the gateway the same
-coding image settings as the server (for example `CODING_WORKER_IMAGE`); the
-gateway only resolves them, it never starts coding workers.
+| Gateway variable               | Purpose                                                                                                                                                                                                                                                    |
+| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `NATIVE_GATEWAY_LISTEN`        | `host:port` to listen on. Default `0.0.0.0:8790`.                                                                                                                                                                                                          |
+| `DATABASE_URL`                 | The same database the server uses. With `DATABASE_AUTH=rds-iam` the gateway also needs `DATABASE_AUTH`, `AWS_REGION` (or `DATABASE_AWS_REGION`) and the CA bundle file named by `sslrootcert` ([details](security-deployment.md#database-authentication)). |
+| LLM provider keys              | The same provider settings as the server (for example `OPENAI_API_KEY`).                                                                                                                                                                                   |
+| `SECRET_APP_KEY`               | Same value as the server, so tools can read their bound secrets or send brokered ones ([details](tool-secrets.md)).                                                                                                                                        |
+| Integration settings           | The same repository-host and issue-tracker settings as the server.                                                                                                                                                                                         |
+| `WARDBY_TOOL_MAX_TIMEOUT_MS`   | Same value as the server, if you set it: the longest time limit a tool may use ([details](security-deployment.md#tool-time-limits)).                                                                                                                       |
+| `WARDBY_MCP_MAX_CALLS_PER_RUN` | Same value as the server, if you set it: the most remote MCP tool calls one coding run may make (default `100`; [details](security-deployment.md#remote-mcp-servers)).                                                                                     |
 
 For local development, the repository ships a compose overlay
 (`deploy/local/docker-compose.native-sandbox.yml`) that runs the gateway next to
@@ -249,7 +254,7 @@ same Wardby image as the server. Requirements:
 - **Environment:** the server's `DATABASE_URL`, `SECRET_APP_KEY` (so tools can
   read their bound secrets or send brokered ones; see
   [tool-secrets.md](tool-secrets.md)), the LLM provider keys, and any integration or
-  coding image settings the server has (see the gateway table in the Docker
+  other settings the server has (see the gateway table in the Docker
   setup above). Also `NATIVE_GATEWAY_LISTEN` (default `0.0.0.0:8790`) and
   `NATIVE_GATEWAY_DENY_PORT` (default `8791`). Keep these in a Secret.
 - **No Kubernetes access:** `automountServiceAccountToken: false` and no Role
@@ -287,13 +292,16 @@ namespace that allows:
 | ----------------------------------- | --------------------------------- |
 | `pods`                              | `create`, `get`, `delete`, `list` |
 | `pods/exec`                         | `create`, `get`                   |
+| `pods/log`                          | `get`                             |
 | `secrets`                           | `create`, `delete`                |
 | `networking.k8s.io/networkpolicies` | `create`, `get`, `delete`         |
 | `services`                          | `get`                             |
 
 `list` on pods lets the startup sweep find leftover worker pods by label;
-`pods/exec` is how the server proves isolation from inside the pod; `services`
-is read once to find the gateway's ClusterIP. Coding runs on Kubernetes use a
+`pods/exec` is how the server proves isolation from inside the pod; `pods/log`
+is read only when a worker exits non-zero, for the redacted tail it logs (without
+it the server logs the exit with no tail); `services` is read once to find the
+gateway's ClusterIP. Coding runs on Kubernetes use a
 largely overlapping Role (`deploy/kind-coding/manifests/base/launcher-role.yaml`).
 
 ### 4. Configure the server
@@ -382,6 +390,22 @@ sandboxing runtime: with this platform the server refuses to start unless
 two minutes to come up, so raise `NATIVE_SANDBOX_READY_TIMEOUT_MS` (for example
 to `600000`).
 
+A run's worker launch (in low-power mode, the gateway wake first, then the pod
+start and the isolation probe) can take several minutes on cold nodes. The
+control plane keeps the run's heartbeat fresh for the whole launch, so the run
+shows as running and the reconciler does not reap it as orphaned. The launch
+is bounded by its own limits instead: the gateway wake
+([`low_power_wake_timeout`](../help/errors/low-power-wake-timeout.md)),
+`NATIVE_SANDBOX_READY_TIMEOUT_MS`
+([`native_sandbox_worker_unready`](../help/errors/native-sandbox-worker-unready.md))
+and the isolation probe
+([`native_sandbox_network_unenforced`](../help/errors/native-sandbox-network-unenforced.md)).
+If the run is stopped during the launch, no worker pod is created
+([`native_sandbox_launch_abandoned`](../help/errors/native-sandbox-launch-abandoned.md)).
+If the control-plane process running the launch exits partway, the run is
+reported [`native_sandbox_worker_lost`](../help/errors/native-sandbox-worker-lost.md)
+and is not relaunched.
+
 ### Capacity
 
 `NATIVE_SANDBOX_MAX_CONCURRENT` caps how many sandbox runs may be active at
@@ -534,6 +558,30 @@ set on the gateway, not the server.
 - A sandbox run has a maximum lifetime of 60 minutes. Past it the worker is
   stopped and the run fails with `native_sandbox_deadline_exceeded`.
 - The agent's own turn and budget limits still apply as for any native agent.
+- Each tool call has the tool's own time limit (`timeoutMs`, default 10
+  seconds, capped by `WARDBY_TOOL_MAX_TIMEOUT_MS`). The worker enforces it; the
+  gateway gives each host call the limit plus 20 seconds, never past the run's
+  deadline. Update the worker image with the server: an older image refuses a
+  run whose agent has a tool with `timeoutMs`
+  ([details](security-deployment.md#tool-time-limits)).
+
+## Stopping a run
+
+`stop_run` (or cancelling the MCP task that started the run) stops a sandboxed
+run whether it is still waiting for a worker or already running:
+
+1. The run is recorded `cancelled` with the caller's reason, unless it already
+   finished (the first terminal state wins; a finished run is not changed).
+2. Its gateway session ends. The worker's next call (a model turn, a tool, or
+   its result) is refused, and the worker exits on its own. A tool's host call
+   still running on the gateway is aborted within a few seconds.
+3. A worker still running 5 seconds later is killed. Either way the worker is
+   then removed; a worker claimed from the warm pool is discarded with its pool
+   row and never returned to the pool.
+4. Sub-agent runs it delegated that are still open are stopped too.
+
+A run stopped before its worker started never starts one. See
+[Stop or cancel a run](../help/stop-run.md).
 
 ## Sub-agents
 
@@ -548,6 +596,32 @@ A child runs in its own configured mode (a `control-plane` child runs in the
 server, a `sandbox` child in its own worker). Cancelling a parent run stops its
 open children.
 
+A coding agent (Codex or Claude Code) can be a child too. The gateway only
+records the delegation. When the server's scheduler leader starts the child, the
+server resolves its worker image (and the Claude Code tool-runner image), reads
+`.wardby/services.yaml` and resolves its services, reads the knowledge bundle
+index, and composes the final task. The gateway therefore needs no `CODING_*`
+settings, and a gateway that sets them can drop them.
+
+A problem with those, such as a `workerImageRef` that is not pinned by digest,
+a toolchain with no image, an invalid or unavailable services declaration, an
+unknown service, or a launcher that cannot run services, shows as the child run
+ending `failed` or `refused` when it starts, with the same error code as at
+dispatch. The parent's `delegate_to_*` call receives the child's result: for a
+failed child `status: "failed"` with the `errorCode`, for a refused child the
+refusal sentence. The full error is on the child run (see it with `get_run`). For agents not
+in sandbox mode these errors still appear at dispatch. If creating a sub-agent
+run fails unexpectedly during a sandboxed run's delegation, the delegating agent
+receives a tool result `{"error":"dispatch_failed","message":"The sub-agent run
+could not be created (<code>)."}` instead of the run failing.
+
+**Upgrading:** upgrade `wardby serve` and `wardby scheduler` before or together
+with the gateway. An older server starts such a child without resolving it: a Codex child
+then runs on the deployment's default Codex image (ignoring its toolchain and
+`workerImageRef`) and without its standing instructions, services, or knowledge
+note, and a Claude Code child fails to start with
+`coding_provider_not_configured:claude-code`.
+
 ## Budget behavior
 
 The gateway is the budget authority for sandboxed runs. Before each model call
@@ -555,28 +629,48 @@ it reserves that call's worst-case cost against the run's budget (and any
 budget group), calls the provider, then settles at the actual usage. A call the
 remaining budget cannot cover is refused and the run ends `budget_exhausted`
 instead of overspending. The worker cannot raise or bypass its budget because it
-never talks to the provider directly.
+never talks to the provider directly. A call that breaks off before its final
+usage is settled at an estimate (never more than its reservation) and
+included in the run's cost. The estimate is the call's input plus the output
+streamed so far (text and tool calls), counted high. If the provider's response
+had started but no output had arrived yet (for example while the model was
+thinking or reasoning), the run's median completed reply is used, or no output
+if there is none yet. A call that failed before the provider's response started is released,
+because nothing was generated.
 
 ## Failure modes
 
-| Error code                                                                                     | Meaning                                                                                                               | What to do                                                                                           |
-| ---------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
-| [`native_sandbox_unavailable`](../help/errors/native-sandbox-unavailable.md)                   | Agent is in `sandbox` mode but the server has no native sandbox. Fails before any spend.                              | Configure the sandbox (steps above) or set the agent back to `control-plane`.                        |
-| [`native_sandbox_worker_exited`](../help/errors/native-sandbox-worker-exited.md)               | The worker ended without producing a result.                                                                          | Check the tool code and the worker's memory and PID limits; run again.                               |
-| [`native_sandbox_deadline_exceeded`](../help/errors/native-sandbox-deadline-exceeded.md)       | The run passed its 60-minute maximum; the worker was stopped.                                                         | Split the work into smaller runs or sub-agents.                                                      |
-| [`native_sandbox_worker_lost`](../help/errors/native-sandbox-worker-lost.md)                   | The worker container or pod disappeared (for example its host or node restarted). It is never relaunched.             | Trigger the run again; check the Docker host or node stability.                                      |
-| [`native_sandbox_image_not_pinned`](../help/errors/native-sandbox-image-not-pinned.md)         | `NATIVE_SANDBOX_WORKER_IMAGE` is a mutable tag.                                                                       | Use a `repo@sha256:...` digest or a local image id.                                                  |
-| [`native_sandbox_docker_failed`](../help/errors/native-sandbox-docker-failed.md)               | A Docker command (pull, network create, connect, run) failed.                                                         | Read the message, then check the Docker daemon, image availability, and gateway name.                |
-| [`native_sandbox_network_unenforced`](../help/errors/native-sandbox-network-unenforced.md)     | Kubernetes: the pod's egress isolation could not be proven in time; the pod was removed.                              | Check that the CNI enforces NetworkPolicy and the gateway Service exposes 8790 and 8791.             |
-| [`native_sandbox_isolation_mismatch`](../help/errors/native-sandbox-isolation-mismatch.md)     | Kubernetes: the stored pod or NetworkPolicy differs from what Wardby built (an admission change).                     | Exempt the run namespace's native-run pods from mutating policies.                                   |
-| [`native_sandbox_gateway_unavailable`](../help/errors/native-sandbox-gateway-unavailable.md)   | Kubernetes: the gateway Service is missing or has no ClusterIP.                                                       | Create the Service (ClusterIP, not headless) or fix `NATIVE_GATEWAY_SERVICE`.                        |
-| [`native_sandbox_worker_unready`](../help/errors/native-sandbox-worker-unready.md)             | Kubernetes: the worker pod did not reach Running in time.                                                             | Check image pull, quotas, and scheduling with `kubectl describe pod`.                                |
-| [`native_sandbox_capacity`](../help/errors/native-sandbox-capacity.md)                         | `NATIVE_SANDBOX_MAX_CONCURRENT` runs are already active, or the namespace ResourceQuota is full. Not queued.          | Wait and retry, or raise the cap together with the ResourceQuota.                                    |
-| [`native_sandbox_warm_delivery_failed`](../help/errors/native-sandbox-warm-delivery-failed.md) | A claimed warm worker could not receive the run's input and could not be confirmed stopped within 30 seconds.         | Trigger the run again; check the node or Docker host and pod exec access.                            |
-| [`native_sandbox_requires_catalog`](../help/errors/native-sandbox-requires-catalog.md)         | No model catalog entry was recorded for the run: the process running it does not use the catalog-backed model router. | Run it on a standard `wardby serve`/`scheduler`/`mcp` process, or keep the agent on `control-plane`. |
+| Error code                                                                                     | Meaning                                                                                                                                                                                                                                                                                                                                                 | What to do                                                                                                                                       |
+| ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
+| [`native_sandbox_unavailable`](../help/errors/native-sandbox-unavailable.md)                   | Agent is in `sandbox` mode but the server has no native sandbox. Fails before any spend.                                                                                                                                                                                                                                                                | Configure the sandbox (steps above) or set the agent back to `control-plane`.                                                                    |
+| [`native_sandbox_worker_exited`](../help/errors/native-sandbox-worker-exited.md)               | The worker ended without producing a result.                                                                                                                                                                                                                                                                                                            | Check the tool code and the worker's memory and PID limits; run again.                                                                           |
+| [`native_sandbox_deadline_exceeded`](../help/errors/native-sandbox-deadline-exceeded.md)       | The run passed its 60-minute maximum; the worker was stopped.                                                                                                                                                                                                                                                                                           | Split the work into smaller runs or sub-agents.                                                                                                  |
+| [`native_sandbox_worker_lost`](../help/errors/native-sandbox-worker-lost.md)                   | The worker container or pod disappeared (for example its host or node restarted). It is never relaunched.                                                                                                                                                                                                                                               | Trigger the run again; check the Docker host or node stability.                                                                                  |
+| [`native_sandbox_image_not_pinned`](../help/errors/native-sandbox-image-not-pinned.md)         | `NATIVE_SANDBOX_WORKER_IMAGE` is a mutable tag.                                                                                                                                                                                                                                                                                                         | Use a `repo@sha256:...` digest or a local image id.                                                                                              |
+| [`native_sandbox_docker_failed`](../help/errors/native-sandbox-docker-failed.md)               | A Docker command (pull, network create, connect, run) failed.                                                                                                                                                                                                                                                                                           | Read the message, then check the Docker daemon, image availability, and gateway name.                                                            |
+| [`native_sandbox_network_unenforced`](../help/errors/native-sandbox-network-unenforced.md)     | Kubernetes: the pod's egress isolation could not be proven in time; the pod was removed.                                                                                                                                                                                                                                                                | Check that the CNI enforces NetworkPolicy and the gateway Service exposes 8790 and 8791.                                                         |
+| [`native_sandbox_isolation_mismatch`](../help/errors/native-sandbox-isolation-mismatch.md)     | Kubernetes: the stored pod or NetworkPolicy differs from what Wardby built (an admission change).                                                                                                                                                                                                                                                       | Exempt the run namespace's native-run pods from mutating policies.                                                                               |
+| [`native_sandbox_gateway_unavailable`](../help/errors/native-sandbox-gateway-unavailable.md)   | Kubernetes: the gateway Service is missing or has no ClusterIP.                                                                                                                                                                                                                                                                                         | Create the Service (ClusterIP, not headless) or fix `NATIVE_GATEWAY_SERVICE`.                                                                    |
+| [`native_sandbox_worker_unready`](../help/errors/native-sandbox-worker-unready.md)             | Kubernetes: the worker pod did not reach Running in time.                                                                                                                                                                                                                                                                                               | Check image pull, quotas, and scheduling with `kubectl describe pod`.                                                                            |
+| [`native_sandbox_capacity`](../help/errors/native-sandbox-capacity.md)                         | `NATIVE_SANDBOX_MAX_CONCURRENT` runs are already active, or the namespace ResourceQuota is full. Not queued.                                                                                                                                                                                                                                            | Wait and retry, or raise the cap together with the ResourceQuota.                                                                                |
+| [`native_sandbox_warm_delivery_failed`](../help/errors/native-sandbox-warm-delivery-failed.md) | A claimed warm worker could not receive the run's input and could not be confirmed stopped within 30 seconds.                                                                                                                                                                                                                                           | Trigger the run again; check the node or Docker host and pod exec access.                                                                        |
+| [`native_sandbox_requires_catalog`](../help/errors/native-sandbox-requires-catalog.md)         | No model catalog entry was recorded for the run: the process running it does not use the catalog-backed model router.                                                                                                                                                                                                                                   | Run it on a standard `wardby serve`/`scheduler`/`mcp` process, or keep the agent on `control-plane`.                                             |
+| [`native_sandbox_gateway_error`](../help/errors/native-sandbox-gateway-error.md)               | The worker's call to the gateway failed with an error the run could not continue from (`native_sandbox_gateway_error: <code>: <message>`; for `internal`, only `native_sandbox_gateway_error: internal: <cause code>`, never the message; the cause code is one of wardby's declared codes and never one that matches a secret value the run can read). | Read the code and message; for `internal` (`internal: <cause code>` when the gateway named one), read the gateway's logs. Trigger the run again. |
 
 A run's error appears in its `error` field in `get_run` and `list_runs`.
 
+For the gateway's `internal` code, the run's error carries a `causeCode` only
+when the failing error's message led with one of wardby's own declared codes,
+the gateway could read the run's secret values, and that code does not match
+any of them; it is shown as `native_sandbox_gateway_error: internal: <causeCode>`.
+Otherwise the error is just `native_sandbox_gateway_error: internal`. The
+gateway's own log line (`native gateway request failed`) carries the error with
+its message and stack, the run's secret values replaced by `[REDACTED]`, so the
+message itself starts with the code; it has no separate `causeCode` field. Only
+when the gateway could not read the run's secret values does the line carry just
+the error's name (`errName`) and a `causeCode` field, with no message.
+See [Native sandbox gateway error](../help/errors/native-sandbox-gateway-error.md).
+
 ## Roll back
 
 Set the agent back to `control-plane` with `update_agent` or
@@ -622,11 +716,29 @@ skipped unless its script sets its flag:
   the Service must list ports 8790 and 8791 and have ready endpoints.
 - **Check a run.** Call `get_run` for the run's status and error code, then use
   the matching error article from `search_help` or `get_help_article`.
+- **A worker exited non-zero.** The server logs `native worker exited non-zero`
+  with the exit code and `stderrTail`, the last 2,000 characters of the worker's
+  output. Before logging it, the server masks the run's gateway capability and
+  every secret value the agent's tools can read as `[REDACTED]`. Encoded forms
+  of a value (URL-encoded, base64) are not matched. Instead of a tail, the line
+  may carry `stderrTailOmitted`: `secrets_unavailable` means one of those
+  secrets could not be read within 10 seconds, so the tail could not be
+  redacted; `log_read_failed` (Kubernetes) means the worker pod's log could not
+  be read within 10 seconds. Either way the run's failure is still recorded. On
+  Kubernetes the tail comes from the last 100 lines of the worker pod's log (the
+  launcher needs `get` on `pods/log`). That read may begin partway through a
+  line (the kubelet splits very long lines, and logs rotate), so its first few
+  dozen characters are always masked, more when an attached secret is long.
 - **Workers cannot reach the gateway.** Confirm `NATIVE_GATEWAY_CONTAINER`
   matches the running container's name exactly and that the gateway listens on
   the port in `NATIVE_GATEWAY_URL` (default 8790).
 - **Sub-agents never start.** Confirm `wardby serve` or `wardby scheduler` is
   running; the scheduler leader starts them.
+- **A tool's console output is missing.** If the gateway logs
+  `sandbox tool console line withheld: the tool's secret values could not be read`,
+  reading one of the tool's attached secrets failed. Without every value it
+  cannot redact the line, so it logs none of it. The tool call itself still
+  succeeds. Check the secret store and the tool's attached secrets.
 - **Tool code fails inside the sandbox.** The worker has no network except the
   gateway and a read-only filesystem apart from `/tmp`; tools must use the
   provided fetch, datastore, and secret functions.

package/docs/observability.md

@@ -49,5 +49,9 @@ the coding-proxy coverage, so verify required signals before production use.
 - Alert on request failures, budget cutoffs, cleanup failures, stalled runs,
   and sustained latency or memory growth.
 - Confirm alerts reach an owned channel and rehearse one response path.
+- Watch the logs for package audit mode: the control plane logs
+  `coding.package_audit.set` when an owner turns it on, off or extends it, and
+  the coding proxy logs `registry.audited` for each package it installs
+  because of it (see [Package audit mode](coding-packages.md#package-audit-mode)).
 - Treat telemetry as operational evidence, not as a replacement for Wardby's
   accounting database.