@namzu/sandbox 15.0.0 → 17.0.0

package/CHANGELOG.md

@@ -1,5 +1,329 @@
 # @namzu/sandbox
 
+## 17.0.0
+
+### Major Changes
+
+- 1722448: **A `container:docker` sandbox with an egress allowlist now needs an internal
+  network, a proxy image, and `container-network` reachability. Take this version
+  if you use one; otherwise nothing you install changes shape.**
+
+  The allowlist tier used to be enforced by `HTTP_PROXY` and nothing else. The
+  proxy ran in the process that created the sandbox, on the host's loopback, and
+  the sandbox kept ordinary bridge networking with full outbound reachability —
+  `--add-host namzu-egress:host-gateway` was the only thing pointing traffic at
+  it. Anything inside the container that opened a socket directly reached the
+  network with the allowlist unconsulted. It is now a sibling container on an
+  `--internal` network the sandbox is also on, which has no route off it: the
+  sandbox's traffic reaches the internet only through that container, and
+  everything else a host attaches to that network is a container on the sandbox's
+  own subnet.
+
+  **What a host using `EgressPolicy` of `static` or `resolver` must now do**, all
+  three refused at `create()` rather than downgraded:
+
+  1. `network` must be an `--internal` network: `docker network create
+--internal <name>`. The network's `Internal` flag is read back from the
+     daemon; the name is never trusted.
+  2. `hostReachability` must be `'container-network'`. A published host port
+     needs a route out and an internal network has none — the refusal names the
+     mode to move to. **A host-side consumer (the CLI, direct dev) that reached
+     the worker on `127.0.0.1:<port>` under an allowlist policy must move to a
+     consumer on the internal network.**
+  3. `egressProxyImage` must name the proxy image:
+
+     ```bash
+     pnpm --filter @namzu/sandbox build
+     docker build -f packages/sandbox/egress-proxy/Dockerfile -t <tag> packages/sandbox
+     ```
+
+     It is a second image because the sandbox image is a string this backend
+     cannot read, and the bind-mount alternative breaks on the remote-daemon
+     deployment `container-network` exists for.
+
+  `HTTP_PROXY` and friends are still set and now direct traffic rather than
+  permit it: a tool that honours them goes through the boundary, one that ignores
+  them fails `Network unreachable` instead of bypassing the policy.
+
+  **Four behaviour changes to know about, all of them consequences of the
+  boundary existing:**
+
+  - A `resolver` policy is resolved at `create()` and at each
+    `setNetworkPolicy()`, not per request. A resolver that rotates between those
+    moments is not picked up until the next one.
+  - `setNetworkPolicy()` replaces the proxy container rather than swapping the
+    allowlist in place. The window between the two has no proxy in it and fails
+    **closed** — no request is permitted that the new policy would refuse.
+  - `brokeredCredentials` are passed to the proxy container's environment. They
+    still never enter the sandbox; they are now readable by anything with access
+    to the docker daemon, which should be treated as credential access. (On the
+    way there the value goes through the `docker` CLI child's ENVIRONMENT, not its
+    argv — an argv is world-readable in `/proc/<pid>/cmdline` on Linux.) Note also
+    that `createSandboxProvider` has never forwarded `brokeredCredentials` at all;
+    that pre-existing gap is unchanged.
+  - `egressProxyUpstreamNetwork` (default `bridge`) is where the proxy reaches
+    the internet. Anything else attached to that network can reach the proxy and
+    use it, so name a dedicated one on a shared daemon. `'none'`, and the internal
+    network itself, are refused: either leaves the proxy with no route out.
+  - Everything else on the SANDBOX's internal network is reachable from the
+    sandbox — a second sandbox, a second sandbox's proxy. That is a property of
+    the network rather than of this tier; give each sandbox its own internal
+    network when they should not see one another.
+
+  `deny-all` and `allow-all` are unaffected beyond the internal network
+  `deny-all` already required. `EgressProxy` gains two optional options
+  (`bindHost`, `selfNames`) and the container entrypoint sets them; the class is
+  otherwise unchanged.
+
+- f1fd331: The container backend's worker now authenticates its caller, and a worker that
+  cannot refuses to start on a routable bind. This is a `major` because the wire
+  shape of a public surface changed: every route but `GET /healthz` now requires
+  `Authorization: Bearer <NAMZU_SANDBOX_TOKEN>`, a missing or wrong token is
+  `401 {"error":"unauthorized"}`, and a worker with no token will not listen on
+  anything but loopback. A token that is set but cannot be answered — empty,
+  padded, or carrying a character an HTTP header cannot hold — now refuses to
+  start rather than leaving a worker that looks authenticated and refuses its own
+  host.
+
+  Nothing in the package's TypeScript API changed — no export was removed,
+  renamed or narrowed, and `HttpWorkerClient` and `execViaHttpWorker` only gained
+  an optional token parameter. What changed is the contract between this backend
+  and the worker process it runs, and that contract is deployed separately: the
+  worker lives in `packages/sandbox/worker/server.js`, which the tarball does not
+  ship, so the image is built from whatever checkout the operator has.
+
+  **What breaks, and for whom.**
+
+  - **A worker image rebuilt from this release, paired with a host that predates
+    it.** The old host injects no token, so the new worker refuses to start on
+    the default `0.0.0.0` bind and every `create()` fails at readiness with the
+    container already exited. The refusal names the variable. Bring the host up
+    in the same step, or set the escape below.
+  - **Anything that talks to the worker without going through this backend.** The
+    route contract is now authenticated; a probe, a health-check script that
+    calls `/execute`, or a hand-rolled client must present the token.
+  - **The standby-pool backend.** A pooled worker built from this release will
+    refuse to boot on its routable default bind, because this backend has no way
+    to hand it a credential. The claim API admits exactly one property override,
+    and it is not `env` — it is a config map, which on Linux reaches the container
+    as a file mount under `/mnt/configmap/<containername>/<key>`, not as an
+    environment variable, while the worker reads its token from `process.env` at
+    startup. The platform does not validate config-map values either, and its own
+    guidance is that a value affecting application security belongs in an
+    environment variable. So the per-instance credential for this backend is NOT
+    implemented here, and the change that would close the gap — a per-claim value
+    in the config map, read by the worker — is written down in
+    `docs/sdk/container-sandbox-worker.md` rather than half-built. A token on the
+    shared profile does not fix it either: the worker would boot and then `401`
+    every call the backend makes, because this backend's client is constructed
+    without a token and it has no field to carry one.
+
+  **A host and a worker from the same release need no configuration.** The
+  container backend mints 32 random bytes per `create()`, hands the value to the
+  docker CLI in its own environment (resolved by a valueless
+  `--env NAMZU_SANDBOX_TOKEN`, so it is in no argv), and sends it as a bearer
+  header on every call it makes. Upgrading both together is the whole migration.
+  The token is per instance, never written to disk, and dies with the container.
+  It is readable where a peer in the same container or a caller who can already
+  talk to the docker daemon could read it — `docker inspect` shows it on the
+  container config for the container's life — which is why it is per-instance
+  rather than shared, and why it is a defence in depth behind network placement
+  rather than a replacement for it.
+
+  **The standby pool: place it first, decide about the credential second.** If you
+  run that backend, the control that actually carries the exposure is the network,
+  and it is the one to get right before anything else — `aci-standby-pool` already
+  refuses to claim a container group without `subnetId` (unless you set
+  `allowPublicAddress: true` and mean it), so the group sits on a private address
+  and that network is what stands in front of the worker. The refusal is unchanged
+  by this release; what has changed is that it is now the first half of the answer
+  rather than the whole of it. The second half is the escape below, which is what
+  makes a pooled worker start at all today: put
+  `NAMZU_SANDBOX_ALLOW_UNAUTHENTICATED=1` on the container group profile, with the
+  group on a private address. It is needed there only because this backend cannot
+  present a credential, and it turns a worker that cannot boot into one that
+  serves inside that address. Read the standby-pool bullet under "What breaks"
+  before taking that step, and note that a token on the shared profile is not an
+  alternative to it.
+
+  **For a deployment the container backend creates**, the escape is a migration
+  tool rather than a destination: it is what keeps you running while the host and
+  the worker image are brought up in the same step.
+
+  **To keep the old behaviour on purpose**, set
+  `NAMZU_SANDBOX_ALLOW_UNAUTHENTICATED=1` in the worker's environment — the
+  container backend's `options.env` for a sandbox it creates, or the standby
+  pool's container group profile for a pooled one. That gives the credential up
+  rather than deferring it: with it set and no token, every route but `/healthz`
+  is open to whoever can route to the container, which is exactly what the
+  default used to be. On the pool, the address comes first and this second; on
+  every other backend, both the host and the image should be upgraded in one step
+  instead. Only `1`, `true`, `yes` and `on` count, in any case and with no
+  surrounding whitespace; everything else — `= 0`, `= false`, `= " yes "` — means
+  off, so the flag cannot turn itself on.
+
+  Also unchanged, and worth repeating from the README: the transport is plain
+  HTTP, so this is a bearer token over the wire and it is defence in depth behind
+  network placement, not a replacement for it. See
+  `docs/sdk/container-sandbox-worker.md`.
+
+### Minor Changes
+
+- 01fbc9b: Three additive declarations, no default changed and nothing removed:
+
+  - `MicroVMBackendConfig.onExecTiming` — the owned Firecracker tier's provider
+    config gains an optional per-exec timing hook.
+  - `FirecrackerTransportTiming` — exported from the package entry point, the
+    shape that hook is called with.
+  - `VsockTransportOptions.onExecTiming` — the same hook at the transport level,
+    for a host that builds a `VsockAgentTransport` itself.
+
+  A host that sets none of them sends, receives and waits for exactly what it did
+  before: the timing accumulator is created only when the hook is set, and the
+  `undefined` checks that skip creating it skip every clock read that would have
+  filled it.
+
+  The hook reports one `exec()`'s wall clock as named phases — `dialMs`,
+  `reserveMs`, `executeMs` and `drainMs`, the four the Kubernetes tier's
+  `onTiming` already reports, plus `firstFrameMs`, `terminatorMs` and
+  `peerCloseMs` for the intervals inside the execute round trip. The three
+  sub-phases are absent, rather than zero, when their phase was never reached; the
+  four base phases are always present and use `0` for "this never happened" (a
+  call that failed at the dial reports `executeMs: 0`). The payload is durations
+  only: never the agent token, a command, its arguments, or any output.
+
+  One behaviour change worth naming, because it is the reason the hook is useful:
+  the transport's execution adapter and its `RemoteExecutionController` are now
+  built per call instead of once per transport. That controller holds no per-call
+  state, so nothing observable changes — but an adapter built once had nowhere
+  per-call to accumulate into, and a single accumulator on the transport would
+  have two concurrent `exec()` calls writing each other's phases. The Kubernetes
+  tier's own transport has been arranged this way since it was written.
+
+  `POST_RESPONSE_CLOSE_TIMEOUT_MS` (1 s) is unchanged and its behaviour is
+  unchanged; it is now documented as what it always was — a reject-only guard
+  that fails a socket whose peer never closes, never a wait a successful call
+  pays. If your host puts a relay between this process and the guest, `peerCloseMs`
+  is the number that tells you whether that relay holds the FIN: a hold **under**
+  a second is reported there on a call that resolved, and a hold **at or past**
+  a second fails the call with `vsock transport: exec peer did not close after
+terminator`, reporting no `peerCloseMs` at all — the close in that case is the
+  one this transport causes itself when the guard fires, and its own constant is
+  not published as an elapsed time. So a resolved exec's fixed cost cannot be
+  hiding in that window: whichever way the window goes, it is either reported or
+  it is a rejection, and never a silent second.
+
+### Patch Changes
+
+- 656598a: Nothing a consumer installs changes. The one edited file is
+  `src/backends/docker/__tests__/leaf-permissions.smoke.test.ts`, which
+  `package.json#files` excludes from the tarball; runtime behaviour, exports, types
+  and defaults are untouched.
+
+  The `Sandbox smoke` workflow has been red on `main` since the docker hardening
+  landed (`481fb8ff`, `--read-only` on by default). That case asserts that uid 1001
+  cannot `mkdir` into the unbound `/mnt/user-data`, and it pinned the refusal to
+  one spelling — `permission denied`. With a read-only rootfs the kernel answers
+  `read-only file system` instead, because EROFS is consulted before the DAC check
+  that would have produced EACCES. The property the case exists for never changed
+  (a bound writable leaf would let the `mkdir` succeed, and the `--rc` assertion
+  still catches that); only the kernel's wording did.
+
+  The assertion now accepts either refusal and says why both are legitimate, so the
+  next change to which check fires first is a one-line read rather than a day of
+  red. `dash` and Docker are both absent from the machine this was written on, so
+  the fix is verified by the workflow that runs this file, not locally.
+
+- Updated dependencies [4e8cf5c]
+- Updated dependencies [19abb2c]
+  - @namzu/sdk@42.0.0
+
+## 16.0.0
+
+### Major Changes
+
+- 11f7ea2: **A `.domain` allowlist entry is expanded on the config-level egress translation, a `specs` list on the applied object is refused, and a deployment that applied the old object must re-apply it before its next `create()`.**
+
+  Everything here is stated against **`@namzu/sandbox@15.0.0`, the published release this one ships on top of** — its bytes as downloaded from the registry, not this branch's base. `15.0.0` emitted the config-level `.domain` entry verbatim and said so itself: its release notes call "that a config-level `.domain` entry matches no answer at all" a pre-existing defect "left exactly as it was and deferred to its own change". This is that change, and the deferral is what the migration below is for.
+
+  `config.egress.policy = { kind: 'static', allowedHosts: ['.example.com'] }` under `engine: 'cilium'` therefore emits `toFQDNs: [{ matchName: example.com }, { matchPattern: '*.example.com' }]` where `15.0.0` emitted `toFQDNs: [{ matchName: '.example.com' }]` — the entry VERBATIM, leading dot included. No DNS answer carries a name with a leading dot, so the old object matched nothing, and nothing refused it on the way in: the shipped admission fence's `matchConditions` scope it to the sandbox host's ServiceAccount, so an operator applying the config-level object is not matched by it at all. `verifyEgressPolicyApplied` then deep-equalled what was sent, the create reported success, and the domain and every subdomain it was asked to allow were DENIED. With `ciliumNarrowing.dnsNames` on it was worse than useless, because the DNS-visibility rule carried the same unmatched names — the DNS proxy denied the lookup the `toFQDNs` half needs before that half could ever learn an address.
+
+  `SandboxNetworkPolicy.allowedHosts` has always said `.example.com` is the domain **and its subdomains** — `packages/sdk/src/types/sandbox/index.ts` — and `src/egress/allowlist.ts` implements exactly that for the docker backend's proxy. The Kubernetes PER-SANDBOX translation already honoured it. The config-level translation now reuses the same helper, so the entry becomes `matchName: example.com` plus `matchPattern: '*.example.com'`, and the DNS-visibility rule carries the names that expansion admits — `matchPattern: '*.<host>.<suffix>'` under every cluster search suffix, which is what the per-sandbox path already emitted.
+
+  **Every consumer-visible change, against `15.0.0`.** Nothing changes for an allowlist with no leading-dot entry: the expansion returns `[{ matchName: entry }]` for an exact host, the DNS rule's wildcard branch is keyed on the leading dot alone, so every other entry's bytes are identical, with and without `dnsNames`, and no entry's letter case is rewritten. The rest is this list, and each item is a create or a compile that behaves differently after the upgrade:
+
+  1. **A config-level `.domain` entry changes the object it emits**, so `verifyEgressPolicyApplied` — which compares the applied object to the translation exactly — refuses your next `createKubernetesWorkspace` or `create()` with `KubernetesEgressPolicyMismatchError` until you **re-apply the manifest this backend now translates**: the same `kubectl apply` you ran for the old one, against a cluster object that admitted nothing. That is the whole migration.
+  2. **A config-level allowlist entry that is not a hostname is refused, by name.** `.com`, `.`, `..example.com`, `*`, `*.example.com`, a URL, a port suffix and an IP literal were each passed straight into the emitted `toFQDNs` by `15.0.0`: the per-sandbox writer validated them, the config-level translation did not. Refused now, with `KubernetesNetworkPolicyHostError` and nothing written, because both halves of what they used to do are wrong — as emitted by `15.0.0` they became a `matchName` that matched nothing, and with the expansion above `.com` would instead emit `matchPattern: '*.com'` and grant every name under a public suffix, silently, on a path no admission policy covers. `.example.com`, `example.com` and `api.example.com` are all still accepted, and a `resolver` policy's returned hosts are held to the same grammar. **A create whose allowlist carries one of these works on `15.0.0` and fails after this upgrade**, until the entry is corrected.
+  3. **`KubernetesNetworkPolicyHostError` loses its third constructor argument.** `15.0.0` ships `constructor(host, reason, options?: { stateHostnameGrammar?: boolean })` on that class, which is exported from the package root; this release ships `constructor(host, reason)`. A consumer that constructed one with three arguments fails to COMPILE (an extra argument is an error in TypeScript), not merely at runtime. The option existed only to suppress one sentence in the config-level refusal, whose reason for existing — a translation that reaches the object as a literal `matchName` admitting nothing — is gone, because the entry expands on both paths now.
+  4. **The `.domain` plus `tlsServerNames` refusal reads as one sentence on both paths.** What is refused is unchanged, and the refusal is still `KubernetesNetworkPolicyHostError` with nothing written; the words changed, because `15.0.0`'s config-level message described a translation that did not expand the entry — which no longer exists. A caller matching that message's text sees new text.
+  5. **A live egress object carrying a `specs` list is refused outright.** A `CiliumNetworkPolicy` carries EITHER one `spec` or a `specs` list, and a rule in either one enforces; the named comparison reads `spec.podSelector`/`spec.endpointSelector`, `spec.policyTypes` and `spec.egress` and nothing else. `15.0.0` read `spec` and reported a match, so an object whose `spec` matched the translation and whose `specs` entry stayed inside the allowlist verified and created — and under `verify: 'named-object-only'`, where nothing else looked at it, so did one whose `specs` entry allowed more. It is a mismatch now, with `KubernetesEgressPolicyMismatchError` naming `specs`. The translation never emits a `specs` list, and the shipped admission fence refuses one for the same reason, so every affected object is a hand-edited one.
+
+  **That re-apply passes, which is not a detail.** Under the default `egress.verify: 'union'` the union check reads the namespace's policies and refuses any that allows more than the translation — and with the expansion in place that has to include the named object, or the fix would refuse the very object it tells an operator to apply: the allowance this check builds records a `toFQDNs` entry by its `matchName` alone, so the expansion's `matchPattern` would read as a widening, permanently, however many times the manifest was re-applied. The check therefore leaves exactly the document the named comparison read — the one built from the object's own `spec` — unjudged, while still reading the named object for the one thing that comparison cannot answer: whether it puts this pod in egress default-deny. So a deployment whose only applied policy is the object this backend names still creates; a second policy — anything else that selects the pod and allows more — is still refused by name; and a `specs` entry is judged like any other object's rules, which is item 5's rule seen from this side.
+
+  **Why `major`, not `patch`.** Bump intent is a claim about the consumer of the version they are on, and against the published `15.0.0` three arguments carry it. (i) A deployment whose config-level allowlist contains a `.domain` entry has a create that succeeds today and fails after this upgrade until an operator re-applies — the emission it depended on was useless, but the failure the upgrade produces is one an operator sees and has to act on. (ii) A deployment whose allowlist contains one of the entries in item 2 has a create that succeeds today and fails with a named refusal after it. (iii) A consumer that constructs the public `KubernetesNetworkPolicyHostError` with its documented third argument no longer compiles, which is a removed part of the public API whatever it did. `@namzu/sandbox`'s own precedent is the same shape: the egress-kinds release was `major` because a deployment with a second policy selecting the sandbox pods began failing `create()` where it used to succeed.
+
+  **Also in this release, and already published in `15.0.0`'s notes rather than here:** `ciliumNarrowing` and the per-sandbox `setNetworkPolicy` policies. Their changeset files were consumed by the `15.0.0` release and are deleted from this branch for that reason — the text lives in the published CHANGELOG, and a pending changeset for it would publish it a second time.
+
+  **Not proven here.** That a Cilium L7 proxy admits `example.com` and its subdomains and nothing else under the emitted pair, and that an uppercase `matchName` (`API.example.com`, emitted exactly as written, as every release has) is admitted by one. This repository has no cluster with a Cilium data plane, and the `kind` cluster its Kubernetes tests use enforces no policy at all, so the translation is pinned and the ENFORCEMENT of it is not.
+
+- 481fb8f: `container:docker` completes its hardening baseline, and a container's root
+  filesystem is now read-only.
+
+  **Before:** the backend passed `--cap-drop=ALL` and
+  `--security-opt=no-new-privileges` and nothing else of its own, so every path
+  inside the container was writable, the container's IPC namespace was whatever
+  the host daemon's `default-ipc-mode` said about sharing it, and CPU was the one
+  resource of the three that had no way to be bounded at all.
+
+  **After:** `--ipc private` and `--read-only` are applied to every container, the
+  four paths that have to stay writable are mounted `--tmpfs` (`/tmp`, `/var/tmp`,
+  `/workspace`, `/home/namzu`, each named with its reason in
+  `src/backends/docker/index.ts`), and a new `cpuLimit` renders `--cpus`.
+
+  **What breaks.** A workload that writes inside the container outside those four
+  paths and the layout's own RW binds — `/opt`, `/srv`, `/etc`, or the `HOME` of an
+  image whose user is not `namzu` — now fails with `EROFS` instead of succeeding.
+
+  A second and much narrower break, in the same class: `--ipc private` makes this
+  container's IPC namespace un-joinable. Docker's `--ipc container:<name>` is gated
+  on the target having a shared-memory directory to enter, and a container created
+  `private` has none — so a container that reached into a sandbox's shared memory,
+  semaphores or message queues that way, which it could because these sandboxes
+  were created `shareable` on any host whose daemon was configured with
+  `default-ipc-mode: shareable`, is now refused by the daemon.
+
+  Scratch also lives in RAM now rather than on the container's writable layer. A
+  temp file larger than half the host's RAM, or larger than `--memory` when the
+  host set one, fails with `ENOSPC` or is OOM-killed, where writing it to disk used
+  to succeed. Any workload that spills more than its memory budget into `/tmp`,
+  `/var/tmp` or `/workspace` is in that group.
+  Everything the reference image does keeps working: `/tmp` stays executable
+  (docker's own `--tmpfs` default is `noexec`, which would have turned
+  `gcc -o /tmp/a.out … && /tmp/a.out` into `Permission denied`), and `HOME` stays
+  writable, which is what LibreOffice, matplotlib and npm need.
+
+  **What to do.** For an image of your own, name what it needs writable in
+  `writableRootfsPaths`; each entry becomes a `--tmpfs`, so it is scratch rather
+  than persistence. For a workload that spills more scratch than its memory
+  budget, move the spill rather than the baseline: `layout.scratch` is a bind to a
+  host directory and is still disk-backed, so give the layout one on a host
+  directory with room and point the workload at it with the per-call `env` option
+  — `TMPDIR` set to that container path — which keeps the read-only root
+  filesystem, the four paths above and the resource bounds. To give the
+  filesystem back — every path inside the container writable again — set
+  `readOnlyRootfs: false`. That turns off that one control: `--ipc private` is
+  applied whatever it says, so what it produces is the argv from before plus that
+  one flag, not the argv from before.
+
+  `cpuLimit` is opt-in and has no default, deliberately: a number chosen here
+  would throttle a run that finishes inside its timeout today, and the right value
+  is a property of the host's machine. It is new surface, and no workload that
+  works today can fail because of it.
+
+  SemVer: **major**, because two defaults changed in ways a working workload can
+  fail under. The additive parts (`cpuLimit`, `writableRootfsPaths`,
+  `readOnlyRootfs`) would be a `minor` on their own.
+
 ## 15.0.0
 
 ### Major Changes

package/README.md

@@ -52,6 +52,7 @@ const provider = createSandboxProvider({
     image: 'namzu-sandbox:latest',
     network: 'namzu-tasks',
     labels: { 'example.task-id': taskId },
+    cpuLimit: 2,
   },
   layout: {
     outputs: { source: { type: 'hostDir', hostPath: `/srv/tasks/${taskId}/outputs` } },
@@ -63,6 +64,117 @@ const provider = createSandboxProvider({
 })
 ```
 
+## Container hardening baseline
+
+`container:docker` confines every container it starts, and the argv it builds is
+pinned by a test (`src/backends/docker/__tests__/hardening.test.ts`) so a change
+to the baseline has to get past it on purpose. Each flag carries its own
+argument in `src/backends/docker/index.ts`; in one line each:
+
+| Flag | Why |
+|---|---|
+| `--cap-drop=ALL` | No Linux capability, with no re-add list. `CAP_DAC_OVERRIDE` alone walks past the layout's read-only binds, and `NET_ADMIN` is what would put a default route back on a `deny-all` network. |
+| `--security-opt=no-new-privileges` | A setuid binary inside the image cannot re-escalate. |
+| `--ipc private` | This container's IPC namespace is not joinable. `shareable` (docker's other daemon default) hands every container a namespace of its own as well, but leaves it joinable by name with `--ipc container:<name>`. |
+| `--read-only` | The image is not a place the workload writes; the layout's `outputs` and `scratch` binds are separate mounts and stay writable. |
+| `--tmpfs <path>` | The paths inside the container that do stay writable — see below. |
+| `--memory`, `--pids-limit`, `--cpus` | Bounds the host sets through `defaultMemoryLimitMb`, `defaultMaxProcesses` and `cpuLimit`. All three are unset by default, because the right number is a property of the host's machine and of the workload. |
+
+**What stays writable under `--read-only`.** Four paths, all `--tmpfs`:
+`/tmp`, `/var/tmp`, `/workspace` and `/home/namzu`. The first three are where a
+workload's own scratch goes — `/tmp` is `TMPDIR`, where pip builds wheels and
+where a program compiled in the sandbox is run, so these mounts are deliberately
+executable (docker's own `--tmpfs` default is `noexec`, which would turn that
+into `Permission denied` on a file that is plainly executable). `/home/namzu` is
+the reference image's `HOME`: LibreOffice refuses a headless conversion without a
+writable user profile, and matplotlib, fontconfig, npm and `pip install --user`
+all keep caches there. A host that points `image` at its own build says what
+that image needs with `writableRootfsPaths` — the backend cannot read an image's
+writable set, and the alternative to asking is guessing. `readOnlyRootfs: false`
+turns that control off — the container filesystem is writable again — and turns
+off nothing else: `--cap-drop=ALL`, `--security-opt=no-new-privileges` and
+`--ipc private` are applied whatever it says.
+
+Because those four paths are tmpfs, they are RAM, not the container's writable
+layer: scratch larger than half the host's RAM (or than `--memory`, the tighter
+of the two when the host sets one) fails with `ENOSPC` rather than spilling onto
+the host's disk. A run that writes temp files bigger than its memory budget does
+not have to give up the baseline over it: `layout.scratch` is a bind to a host
+directory and stays disk-backed, so a host with room on disk mounts one there
+and points the workload at it — `TMPDIR` set to that container path through the
+per-call `env` option — which keeps the read-only root filesystem and the four
+paths above. `readOnlyRootfs: false` is the last resort rather than the first:
+it buys the container's own writable layer back at the cost of the control.
+
+**Three controls are deliberately absent.** A seccomp profile: docker applies
+its built-in one to every container and nothing here asks for anything looser,
+so what is missing is a *tighter* profile, and a hand-written one cannot be
+verified here against the reference image's toolchain — a profile that blocks a
+syscall chromium or LibreOffice needs breaks the sandbox, which is worse than
+the gap it closes. Set `seccomp-profile` in the daemon's `daemon.json` if you
+want one. `--userns-remap`: it is a daemon property (`userns-remap` in
+`daemon.json`), not a `docker run` flag — a container only chooses between the
+namespaces the daemon already made (`--userns=host|private`), so whether a
+remapped mapping exists is settled before this backend's argv is read and no
+flag here could settle it. Enable it on the host and every container in this
+tier gets uid 0 mapped to an unprivileged uid outside.
+`--user`: supported through `runAsUser` and unset by default, because `--user`
+overrides the image's own choice and the reference image already ends with
+`USER namzu`.
+
+## Container egress boundary
+
+An egress policy of `static` or `resolver` — a host allowlist — is enforced by
+the egress proxy running as a **sibling container**, not by the process that
+created the sandbox. The proxy is dual-homed: `docker run` puts it on an
+ordinary network so it has a route to the internet, and `docker network
+connect --alias namzu-egress` adds the `--internal` network the sandbox is on.
+The sandbox joins that internal network alone and has no route off it — a route
+it cannot put back, because `--cap-drop=ALL` removed `NET_ADMIN`.
+`--add-host namzu-egress:host-gateway` and the loopback proxy that needed it are
+gone.
+
+"No route off it" is the accurate half of that, and it is worth being exact
+about the other: the internal network is a subnet, so the sandbox can also
+reach whatever else a host attaches there — a second sandbox, a second
+sandbox's proxy. Give each sandbox its own internal network when they should
+not see one another; see
+[docs/sdk/sandbox-egress.md](../../docs/sdk/sandbox-egress.md).
+
+`HTTP_PROXY`, `http_proxy`, `HTTPS_PROXY`, `https_proxy` and `NO_PROXY` are
+still set on the sandbox, and what they are has changed. They no longer permit
+traffic, they direct it: a tool that honours them sends its request through the
+boundary, and a tool that ignores them — a binary that does not read proxy
+environment, `curl --noproxy '*'`, a raw socket — has nowhere to send anything
+and fails with `Network unreachable`. Before this, that second tool reached the
+network with the allowlist unconsulted.
+
+Three things a host must supply, each refused at `create()` rather than
+downgraded: an `--internal` network in `network` (`docker network create
+--internal <name>`), `hostReachability: 'container-network'` (a published host
+port needs a route out that this network does not have), and
+`egressProxyImage` — the proxy image, built from
+`packages/sandbox/egress-proxy/Dockerfile` exactly as the sandbox image is
+built from `packages/sandbox/worker/Dockerfile`. Nothing in this repository
+pushes an image; both are built by hand and named by tag in the config.
+
+The proxy container carries the same baseline the sandbox does
+(`--cap-drop=ALL`, `--security-opt=no-new-privileges`, `--ipc private`,
+`--read-only`), because it is the process standing between untrusted code and
+the internet. `deny-all` and `allow-all` need none of this beyond the internal
+network `deny-all` already required.
+
+What the boundary does not cover — domain fronting inside a `CONNECT` tunnel, a
+`resolver` policy resolved at `create()` and `setNetworkPolicy()` rather than
+per request, `setNetworkPolicy()` replacing the proxy container rather than
+swapping a list in place, brokered credentials now readable by anything with
+daemon access, the proxy's listener being reachable by whatever else shares its
+upstream network, and everything else the sandbox shares its own internal
+network with — is stated in
+[docs/sdk/sandbox-egress.md](../../docs/sdk/sandbox-egress.md), along with the
+argv-level evidence this change is verified by and the fact that no test here
+starts a container.
+
 ## Protocol readiness and cancellation
 
 Pass `SandboxExecOptions.signal` to stop a command on any shipped backend. A
@@ -522,6 +634,117 @@ sees the agent's own settings — `NAMZU_SANDBOX_WORKSPACE` among them. A
 workload that needs a value in its terminal passes it in `env` on the
 `openTerminal` call, which still wins over everything else, `TERM` included.
 
+## The container worker's control API, and its token
+
+The container backend runs `worker/server.js` — a different process from the
+guest agent above, with a different wire: plain HTTP on a port Docker forwards,
+not a framed stream over a socket. It serves `GET /healthz`,
+`POST /execute`, `POST /executions/reserve`, `POST /cancel`, `POST /read-file`
+and `POST /write-file`, and until recently it authenticated nothing: any peer
+that could route to the container could run a command or read and write a file
+inside it. What made that defensible was the network the container is attached
+to, which is a property of a deployment rather than of the worker, and is
+absent wherever a control plane can put the container on a public address.
+
+Every route but `GET /healthz` now requires
+`Authorization: Bearer <NAMZU_SANDBOX_TOKEN>`. A missing, empty or different
+token is answered `401` with `{"error":"unauthorized"}` and nothing else — no
+expected/got, no length, no hint about which routes exist — and the request
+never reaches a handler, so a refused `/write-file` writes nothing. The
+comparison is a fixed-width SHA-256 digest compare, so neither the value nor
+its length is learnable by probing, and the gate runs before every dispatch,
+including the 404, so an unauthenticated caller cannot tell a real route from a
+missing one, or a wrong token from a missing one. `/healthz` requires no token
+and never echoes one, because it is what the host polls before it has any other
+business with the worker, and it answers with liveness and the protocol version
+only. That exemption is an exact match on the whole URL, so `POST /healthz`,
+`GET /healthz?x=1` and `GET /healthz/` are gated like anything else. It also
+leaves exactly one bit readable without a credential, deliberately: the
+retiring-worker check answers before the `/healthz` dispatch, so a worker that
+has poisoned itself answers `503 {"error":"worker_retiring"}` where a healthy
+one answers `200`. That is the drain signal, and the readiness probe — which
+has no credential to present — is who reads it; on every route that does
+anything, a caller without the token gets the same `401` from a retiring worker
+and a serving one.
+
+**The token is minted per instance, by whoever starts the container.** The
+container backend generates 32 random bytes at `create()` time and hands the
+value to the docker CLI in ITS environment, resolved by the valueless
+`--env NAMZU_SANDBOX_TOKEN`. It shares a destination with three variables that
+already exist — the workspace path and the read/write roots land in the
+container's environment through the same `--env` mechanism — and nothing else:
+those three are rendered in the argv as `--env K=V`, values and all, while this
+one is valueless, which is docker's form for "take the value from the CLI's own
+environment". So it is in no argv, `ps` on the host does not show it, and a
+failed `docker run` renders its argv with every `--env` value redacted, in all
+four spellings of the flag, so the error a host logs does not carry it either.
+What it IS visible in, said plainly: the container's own config, so
+`docker inspect <name>` shows it for the container's life to anyone who can
+already talk to the daemon, and the worker's `/proc` inside the container to a
+workload sharing its uid. That is why it is per-instance and why it dies with
+the container — an image-level secret would be shared by every container ever
+built from it and readable by anything that can pull it, and a profile-level one
+would be shared by every instance claimed from a pool. The variable carries the
+`NAMZU_SANDBOX_` prefix, which is what keeps it out of every command the sandbox
+runs: the worker strips that prefix from the environment it hands to a spawned
+command, so a sandboxed task cannot read the credential out of its own
+environment.
+
+**A worker with no token refuses to start on a routable address.** The bind
+default is `0.0.0.0` and stays that way: a published container port forwards to
+the container's interface address rather than to its loopback, so narrowing the
+default disables the container backend instead of hardening it. The credential
+is what makes that default defensible, so its absence fails closed:
+
+| What the worker was given | What it does |
+|---|---|
+| `NAMZU_SANDBOX_TOKEN` set | Requires it on every route but `/healthz`, whatever the bind address |
+| No token, bound to loopback (`127.0.0.1`, `::1`, `localhost`) | Starts, unauthenticated. Nothing outside the container's own network namespace can open that socket, and refusing here would break the host-beside-docker dev case while closing nothing |
+| No token, bound anywhere else — including the `0.0.0.0` default | Refuses to start, exit 1, naming the variable and every way out |
+| `NAMZU_SANDBOX_TOKEN` set but **empty**, or with leading/trailing whitespace | Refuses to start in every mode. An empty value is the shape an injected secret takes when the injection resolved to nothing; a padded one is read from a trimmed header and so can never be presented, leaving a worker that looks authenticated and refuses its own host for the container's life |
+| `NAMZU_SANDBOX_TOKEN` set to something **no HTTP header can carry** — a code point above U+00FF, or a C0 control other than HTAB | Refuses to start in every mode. A header value is one byte per character, so the client's own `fetch` throws before the request leaves the host for the first, and the HTTP parser on this side drops the connection for the second. Same "boots authenticated, refuses its host" state, caught at boot instead of at the first call |
+| No token, routable, and `NAMZU_SANDBOX_ALLOW_UNAUTHENTICATED=1` | Starts, unauthenticated, on purpose |
+
+**The escape, and what it gives up.**
+`NAMZU_SANDBOX_ALLOW_UNAUTHENTICATED=1` is the explicit, named way to keep an
+existing deployment running unauthenticated. It gives up the credential
+entirely rather than deferring it: every route but `/healthz` is then open to
+whoever can route to the container, which is exactly the pre-token behaviour
+and exactly what the credential was added to remove. Only `1`, `true`, `yes` and
+`on`, in any case and with no surrounding whitespace, turn it on: `= 0` and
+`= false` mean off, and so does everything else, because a flag whose
+off-spelling turns it on is a trap — and so is a security escape that accepts
+the shape a value takes when an editor adds a space to it. A configured token
+always wins over the escape, since the escape can only mean "serve without a
+credential", never "ignore the one I was given".
+
+**A worker this host did not create must be provisioned with the token by
+whoever does.** There is no channel back: the worker is handed its credential
+at startup and never publishes it, so a warm pool, a shared container-group
+profile or a container someone else started cannot be authenticated against by
+a host that has no way to learn what it holds. The standby-pool backend cannot
+send a per-instance token: the one property override its claim API admits is a
+config map, and a config map arrives as a file mount under `/mnt/configmap`,
+not as an environment variable — while this worker reads its token from
+`process.env` at startup and never looks at the filesystem for one. That is a
+channel that exists and a credential that still does not go in it, for two
+reasons rather than none: the worker would not read it, and Microsoft's own
+guidance is that config map values are not validated by the runtime and are not
+where a value affecting application security belongs. A token on the shared
+profile would be one credential for every instance claimed from the pool, and
+this backend has no field to present one anyway — so what runs there today is a
+private address (`subnetId`) plus the worker-side escape. The full answer, and
+the change that would close the gap, are in
+`docs/sdk/container-sandbox-worker.md`.
+
+**The transport is not confidential.** `Authorization: Bearer` over plain HTTP
+is replayable by anything on the path, so this is defence in depth behind
+network placement, not a replacement for it. On the container backend the
+worker is reached over Docker's port-forward on host loopback, or by DNS name
+on a private bridge; it does not make a public-address deployment safe, which is
+why the standby-pool backend still refuses to claim one without `subnetId`. The
+egress proxy is unrelated to any of this and checks nothing inbound.
+
 ## Firecracker workspace channels
 
 The Firecracker backend exposes two optional same-sandbox channels. Call

package/dist/backends/aci-standby-pool/index.d.ts

@@ -132,10 +132,28 @@ export declare function assertEnforceable(options: SandboxBackendOptions): void;
  * so nothing looks wrong. A caller who never heard of `subnetId` gets a
  * working sandbox on the internet and no signal at all.
  *
- * What is on that address matters: `worker/server.js` states "Authn: none"
- * in its own docblock and binds every interface. Inside a private network
- * that is the boundary doing the work. With a public address there is no
- * boundary left, and the worker's `/execute` is reachable by anyone.
+ * What is on that address matters: `worker/server.js` binds every interface
+ * and, inside a private network, that was the boundary doing the work — the
+ * worker authenticated nobody. With a public address there is no boundary
+ * left, and the worker's `/execute` is reachable by anyone.
+ *
+ * The worker now requires a per-instance `Authorization: Bearer` token on
+ * every route but `/healthz`, and REFUSES TO START on a routable bind when
+ * it has none. This backend cannot supply one, and the reason this comment
+ * used to give for that was wrong: it said the claim API's single admitted
+ * override — a config map — is no channel into the container group. It is
+ * one, on Linux a file mount under `/mnt/configmap/<containername>/<key>`
+ * carrying caller-supplied per-claim values. The credential is declined
+ * anyway, for two reasons that survive their source: the worker reads its
+ * token from `process.env` at startup and never looks for a file, so a
+ * mounted value is not read; and Microsoft's own guidance is that config
+ * map values are not validated by the runtime and that values affecting
+ * application security "should be made available to the container using
+ * environment variables". The full answer, and the change that would close
+ * the gap, is in `docs/sdk/container-sandbox-worker.md`. What that means
+ * for an operator is written down there too, rather than left to be
+ * discovered at startup; this refusal is unchanged, and now stands in
+ * front of a worker that would refuse the routable case itself.
  *
  * Defaulting to refusal rather than to a warning, because a warning on a
  * path that otherwise succeeds is read once and never again.

package/dist/backends/aci-standby-pool/index.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../../src/backends/aci-standby-pool/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAwCG;AAEH,OAAO,KAAK,EAEX,8BAA8B,EAU9B,MAAM,YAAY,CAAA;AAGnB,OAAO,KAAK,EAAE,cAAc,EAAE,qBAAqB,EAAE,MAAM,gBAAgB,CAAA;AAc3E;;;;;GAKG;AACH,MAAM,MAAM,gBAAgB,GAAG,MAAM,OAAO,CAAC,MAAM,CAAC,CAAA;AAEpD,MAAM,WAAW,mCAAmC;IACnD,QAAQ,CAAC,cAAc,EAAE,MAAM,CAAA;IAC/B,QAAQ,CAAC,aAAa,EAAE,MAAM,CAAA;IAC9B,QAAQ,CAAC,QAAQ,EAAE,MAAM,CAAA;IACzB;;;;OAIG;IACH,QAAQ,CAAC,qBAAqB,EAAE,MAAM,CAAA;IACtC;;;OAGG;IACH,QAAQ,CAAC,+BAA+B,EAAE,MAAM,CAAA;IAChD;;;OAGG;IACH,QAAQ,CAAC,6BAA6B,CAAC,EAAE,MAAM,CAAA;IAC/C;;;OAGG;IACH,QAAQ,CAAC,MAAM,EAAE,8BAA8B,CAAA;IAC/C;;OAEG;IACH,QAAQ,CAAC,WAAW,EAAE,gBAAgB,CAAA;IACtC;;;;;;OAMG;IACH,QAAQ,CAAC,QAAQ,CAAC,EAAE,MAAM,CAAA;IAC1B;;;;;;;;;;;;;OAaG;IACH,QAAQ,CAAC,kBAAkB,CAAC,EAAE,OAAO,CAAA;IACrC,QAAQ,CAAC,mBAAmB,CAAC,EAAE,MAAM,CAAA;IACrC,QAAQ,CAAC,cAAc,CAAC,EAAE,MAAM,CAAA;IAChC;;OAEG;IACH,QAAQ,CAAC,UAAU,CAAC,EAAE,MAAM,CAAA;IAC5B,QAAQ,CAAC,aAAa,CAAC,EAAE,MAAM,CAAA;IAC/B;;;;;;OAMG;IACH,QAAQ,CAAC,mBAAmB,CAAC,EAAE,MAAM,CAAA;CACrC;AASD;;;;GAIG;AACH,wBAAgB,0BAA0B,CACzC,MAAM,EAAE,mCAAmC,GACzC,cAAc,CAiBhB;AAoMD,wBAAgB,iBAAiB,CAAC,OAAO,EAAE,qBAAqB,GAAG,IAAI,CActE;AAED;;;;;;;;;;;;;;;;GAgBG;AACH,wBAAgB,0BAA0B,CAAC,MAAM,EAAE;IAClD,QAAQ,CAAC,EAAE,MAAM,CAAA;IACjB,kBAAkB,CAAC,EAAE,OAAO,CAAA;CAC5B,GAAG,IAAI,CAOP"}
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../../src/backends/aci-standby-pool/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAwCG;AAEH,OAAO,KAAK,EAEX,8BAA8B,EAU9B,MAAM,YAAY,CAAA;AAGnB,OAAO,KAAK,EAAE,cAAc,EAAE,qBAAqB,EAAE,MAAM,gBAAgB,CAAA;AAc3E;;;;;GAKG;AACH,MAAM,MAAM,gBAAgB,GAAG,MAAM,OAAO,CAAC,MAAM,CAAC,CAAA;AAEpD,MAAM,WAAW,mCAAmC;IACnD,QAAQ,CAAC,cAAc,EAAE,MAAM,CAAA;IAC/B,QAAQ,CAAC,aAAa,EAAE,MAAM,CAAA;IAC9B,QAAQ,CAAC,QAAQ,EAAE,MAAM,CAAA;IACzB;;;;OAIG;IACH,QAAQ,CAAC,qBAAqB,EAAE,MAAM,CAAA;IACtC;;;OAGG;IACH,QAAQ,CAAC,+BAA+B,EAAE,MAAM,CAAA;IAChD;;;OAGG;IACH,QAAQ,CAAC,6BAA6B,CAAC,EAAE,MAAM,CAAA;IAC/C;;;OAGG;IACH,QAAQ,CAAC,MAAM,EAAE,8BAA8B,CAAA;IAC/C;;OAEG;IACH,QAAQ,CAAC,WAAW,EAAE,gBAAgB,CAAA;IACtC;;;;;;OAMG;IACH,QAAQ,CAAC,QAAQ,CAAC,EAAE,MAAM,CAAA;IAC1B;;;;;;;;;;;;;OAaG;IACH,QAAQ,CAAC,kBAAkB,CAAC,EAAE,OAAO,CAAA;IACrC,QAAQ,CAAC,mBAAmB,CAAC,EAAE,MAAM,CAAA;IACrC,QAAQ,CAAC,cAAc,CAAC,EAAE,MAAM,CAAA;IAChC;;OAEG;IACH,QAAQ,CAAC,UAAU,CAAC,EAAE,MAAM,CAAA;IAC5B,QAAQ,CAAC,aAAa,CAAC,EAAE,MAAM,CAAA;IAC/B;;;;;;OAMG;IACH,QAAQ,CAAC,mBAAmB,CAAC,EAAE,MAAM,CAAA;CACrC;AASD;;;;GAIG;AACH,wBAAgB,0BAA0B,CACzC,MAAM,EAAE,mCAAmC,GACzC,cAAc,CAiBhB;AAoMD,wBAAgB,iBAAiB,CAAC,OAAO,EAAE,qBAAqB,GAAG,IAAI,CActE;AAED;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAkCG;AACH,wBAAgB,0BAA0B,CAAC,MAAM,EAAE;IAClD,QAAQ,CAAC,EAAE,MAAM,CAAA;IACjB,kBAAkB,CAAC,EAAE,OAAO,CAAA;CAC5B,GAAG,IAAI,CAOP"}