@mseep/open-computer-use 1.0.0

package/docs/future-architecture/architecture/01-layers.md

@@ -0,0 +1,109 @@
+<!-- SPDX-License-Identifier: FSL-1.1-Apache-2.0 -->
+<!-- Copyright (c) 2025 Open Computer Use Contributors -->
+
+# 01 — The 4-Layer Model
+
+> Target architecture, adapted from internal design notes.
+> Same model. Same separation of concerns. Our concrete component names.
+
+## Diagram
+
+```text
+┌─────────────────────────────────────────────────────────────────────┐
+│  LAYER 4 — Control Plane                          (Go service)      │
+│  • User-facing API: MCP gateway + REST/GraphQL for admin UI         │
+│  • Auth: OIDC / JWT, tenancy, RBAC                                  │
+│  • Session router: session_id → sandbox handle (KV store)           │
+│  • Secret broker: short-lived creds, key rotation                   │
+│  • Quota / rate-limit / audit log                                   │
+│  • Egress proxy management (JWT-allowlist signing)                  │
+└────────────────────────┬────────────────────────────────────────────┘
+                         │   HTTP / gRPC (internal, mTLS)
+                         ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│  LAYER 3 — Orchestrator / Provider             (pluggable)          │
+│  SandboxProvider interface:                                         │
+│    spawn(template) → handle    exec(handle, cmd) → stream           │
+│    configure(handle, ctx)      stop(handle)                         │
+│    list() / health(handle)                                          │
+│                                                                     │
+│  Implementations:                                                   │
+│    • DockerComposeProvider  ← PoC, current path (Phases 1–4)        │
+│    • KubernetesProvider     ← prod, on agent-sandbox CRDs (Phase 5) │
+│    • DirectCHProvider       ← bare-metal microVM (Phase 9+, opt)    │
+│                                                                     │
+│  Owns: scheduling, warm pool, networking, storage binding           │
+└────────────────────────┬────────────────────────────────────────────┘
+                         │   creates / drives
+                         ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│  LAYER 2 — Sandbox Runtime                     (pluggable)          │
+│  Selected per template via RuntimeClass / direct hypervisor:        │
+│    • runc       — dev/CI, no isolation                              │
+│    • sysbox     — internal/trusted, fast, kernel-shared             │
+│    • gVisor     — code-exec only (NOT browser)                      │
+│    • kata-ch    — Cloud Hypervisor microVM, untrusted Computer Use  │
+│    • kata-fc    — Firecracker microVM, fastest cold start           │
+└────────────────────────┬────────────────────────────────────────────┘
+                         │   provides PID 1 process namespace
+                         ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│  LAYER 1 — Guest Agent                                              │
+│  Today:  Python entrypoint + MCP server inside image  (transition)  │
+│  Future: small Rust static binary as PID 1          (Phase 7)       │
+│  Surface: data-plane WS (vsock on microVM, TCP elsewhere)           │
+│         + control-plane HTTP (health/shutdown, fs_freeze Phase 10)  │
+│  Duties: exec, file ops, port-forward, CDP proxy, ttyd, MCP tools   │
+│  Does NOT: authenticate (L4 does), persist state (L3 does)          │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+## Why this split
+
+- **Independent evolution.** Replace the runtime (L2) without touching the agent (L1) or the orchestrator (L3). Add a new orchestrator (L3) without changing the user-facing protocol (L4).
+- **Threat-model-driven runtime choice.** Same agent, same control plane, different L2 for different tenants. Trusted internal → sysbox. Public Computer Use → Kata + Cloud Hypervisor.
+- **One protocol for users.** L4 exposes **MCP** (already in production with us) plus a thin admin REST/GraphQL. L1–L3 internal contracts stay internal.
+
+## Mapping today's code to this model
+
+| Today | Where it lives | Future layer | Migration phase |
+|---|---|---|---|
+| `computer-use-server/app.py` (FastAPI, MCP, uploads, auth) | repo root | **L4** Control Plane (will be Go) | Phase 6 cutover |
+| `computer-use-server/docker_manager.py` (Docker socket, lifecycle, cleanup) | repo root | **L3** Provider (`DockerComposeProvider`) | Phase 1 extract behind interface |
+| `computer-use-server/mcp_tools.py` (bash/python/file tools) | repo root | **L4** (gateway) + **L1** (exec target) | Phase 7 split |
+| `Dockerfile` entrypoint, in-image MCP server | sandbox image | **L1** Guest Agent (Python → Rust per [ADR-0002](../adr/0002-guest-agent-language-go.md)) | Phase 7 |
+| Docker (`runc`) as runtime | host | **L2** Runtime (`runc` tier) | Phase 9 adds Kata tiers |
+| `helm/computer-use-server/` (single Deployment + DinD sidecar) | repo | **L3** + **L4** k8s manifests, split | Phases 5, 6 |
+| `/tmp/computer-use-data` + Docker volumes | host fs | **Storage** ([06-storage.md](./06-storage.md)) — moves to S3 | Phase 3 |
+| Static env-var secrets (Anthropic, GitLab, vision) | container env | **L4** secret broker | Phase 4 |
+
+## What changes for users — nothing (intentionally)
+
+- The MCP contract (tools, headers, auth) stays stable across every phase.
+- Docker Compose PoC keeps working through Phase 10.
+- Open WebUI integration is L4-facing — unchanged.
+
+## What does NOT belong to any of these layers
+
+- **Skills** (the AI capability bundles under `skills/`) — packaging, not architecture. They mount into L1 sandboxes. See [06-storage.md](./06-storage.md).
+- **Open WebUI** — a downstream consumer of L4's MCP gateway. Not part of this stack.
+- **Sub-agent CLIs** (claude, codex, opencode) — executed *inside* L1. Tooling, not architecture.
+
+## Reference architectures we draw from
+
+- **AWS Lambda** ([`references.md`](../references.md) Lambda framing, [ADR-0010](../adr/0010-lambda-as-inspiration-not-runtime.md)) — pattern source for Firecracker tiering, two-tier control split, and snapshot-pool cold-start economics. **Inspiration, not deployment substrate.**
+- **E2B `envd`** ([`research/02`](../research/02-e2b-infra.md)) — Go-language L1 reference and production-shape validation.
+- **Coder** ([`research/03`](../research/03-coder.md)) — workspace-proxy and multi-region patterns for Phase 10.
+- **bubblewrap / seatbelt sandboxing** (industry-observed) — secondary-defense patterns inside the guest that inform hardening within microVMs at Phase 9.
+
+## Source
+
+- Internal design notes — the layered model and shared vocabulary.
+
+## See also
+
+- [02 — Layer 4: Control Plane](./02-layer4-control-plane.md)
+- [03 — Layer 3: Providers](./03-layer3-providers.md)
+- [04 — Layer 2: Runtimes](./04-layer2-runtimes.md)
+- [05 — Layer 1: Guest Agent](./05-layer1-guest-agent.md)
+- [Roadmap](../roadmap.md)

package/docs/future-architecture/architecture/02-layer4-control-plane.md

@@ -0,0 +1,122 @@
+<!-- SPDX-License-Identifier: FSL-1.1-Apache-2.0 -->
+<!-- Copyright (c) 2025 Open Computer Use Contributors -->
+
+# 02 — Layer 4: Control Plane
+
+> Status: **design** (locked target). Implementation lands in roadmap Phase 6.
+> Language: **Go** ([ADR-0001](../adr/0001-control-plane-language-go.md)).
+> Until Phase 6 ships, today's `computer-use-server/` (FastAPI) **is** the de-facto L4 — Phases 1–5 evolve it in place.
+
+## Responsibilities
+
+1. **User-facing MCP gateway.** Accept MCP JSON-RPC over HTTP/WebSocket (the same surface today's `app.py` exposes at `/mcp`). Authenticate, route to a session's L1 agent, stream results back.
+2. **Admin REST/GraphQL.** Operator-facing API: list sessions, drain a tenant, rotate keys, push a new `SandboxTemplate`, view audit log. Backs the admin UI.
+3. **Tenancy & auth.** OIDC (employee + customer), JWT issuance for sandbox-internal use, RBAC.
+4. **Session router.** `session_id → { sandbox_handle, tenant_id, template_id, created_at }`. Backed by a fast KV (Redis / Valkey / etcd). Single source of truth for "which sandbox serves this chat".
+5. **Secret broker.** Mint short-lived, scoped credentials (Anthropic, GitLab, S3 STS) on session start; rotate without restarting the sandbox. See [07-security.md](./07-security.md).
+6. **Egress JWT signer.** Issue per-session JWTs that encode allowed egress destinations; the egress proxy ([09-templates.md](./09-templates.md) + [08-networking.md](./08-networking.md)) validates them.
+7. **Quota / rate-limit.** Per-tenant concurrent sandboxes, per-tenant request rate.
+8. **Audit log.** Structured events: session created/destroyed, template assigned, exec called, egress request, secret rotated. Retention ≥ 90 days. See [10-observability.md](./10-observability.md).
+
+## What L4 must NOT do
+
+- **Spawn sandboxes directly.** It calls L3 providers — never `docker.run` or `kubectl apply`.
+- **Hold long-lived sandbox credentials in env.** Secret broker mints them per-session.
+- **Trust L1 agents.** L1 is reachable only through L3-managed network paths; L4 ↔ L3 is the only authenticated hop.
+
+## API surface (target)
+
+| Endpoint | Purpose | Notes |
+|---|---|---|
+| `POST /mcp` | MCP JSON-RPC (initialize / tools/list / tools/call) | Bearer auth; existing contract preserved |
+| `GET  /mcp/sse` or `WS /mcp` | Streaming for long tool calls | Today: synchronous HTTP; future: streaming |
+| `GET  /healthz`, `/readyz` | K8s probes | Same as today |
+| `POST /api/uploads` / `GET /api/files/{path}` | Per-tenant user data I/O | Backed by S3 (Phase 3+) |
+| `GET  /system-prompt` | Tenant-scoped system prompt rendering | Same as today |
+| `POST /admin/tenants/{id}/keys/rotate` | Force-rotate tenant secrets | Admin-only |
+| `GET  /admin/sessions` | List + filter | Admin-only |
+| `POST /admin/sessions/{id}/terminate` | Force kill | Admin-only |
+| `GET  /admin/templates` / `POST /admin/templates` | CRUD sandbox templates | Admin-only; see [09-templates.md](./09-templates.md) |
+| `GET  /admin/audit` | Query audit log | Read-only, append-only store |
+
+The `POST /mcp` contract is **frozen** as the user-facing surface — never break it across phases.
+
+## Internal contracts
+
+- **L4 → L3 (provider):** **connect-go** (gRPC + Connect + HTTP/JSON from one `.proto`), mTLS in production. See [ADR-0008](../adr/0008-internal-grpc-external-rest-mcp.md). Operations: `Spawn(template, tenant_ctx)`, `Configure(handle, ctx)`, `Exec(handle, cmd) → stream<Output>`, `Stop(handle)`, `List(filter) → stream<Handle>`, `Health(handle)`, `Events() → stream<Event>`. See [03-layer3-providers.md](./03-layer3-providers.md).
+- **L4 → secret stores:** AWS Secrets Manager / Vault / k8s `Secret` — read at startup + on rotation. Never embedded in container images.
+- **L4 → KV (session store):** Redis / Valkey / etcd. Failure mode: lose recent session routing → sessions reconnect (sessions are short-lived). Snapshot for multi-AZ.
+- **L4 ↔ user UI (CDP / ttyd):** **WebSocket passthrough** — L4 does **not** parse CDP messages; it consistent-hashes by session ID and shovels frames bidirectionally. Same path for ttyd. See [ADR-0008](../adr/0008-internal-grpc-external-rest-mcp.md) for why.
+
+## External surface — protocol map
+
+| Caller | Protocol | Reason |
+|---|---|---|
+| User agents / Open WebUI | **MCP** (JSON-RPC over HTTP/WebSocket) | Frozen contract ([ADR-0005](../adr/0005-mcp-as-control-plane-gateway.md)) |
+| Admin UI | **REST** (OpenAPI-described) | Standard for SPAs, browser-debuggable |
+| User UI CDP / ttyd | **WebSocket** passthrough | Long-lived binary streams, opaque to L4 |
+| Internal L3 / pool-manager | **connect-go** mTLS | Schema-first, typed, streaming-native |
+
+MCP semantics live **only** in the gateway layer of L4. It translates MCP `tools/call` into typed connect-go calls on the provider. L1 agents do not speak MCP — they speak connect-go.
+
+## Admin UI
+
+- **Scope (MVP, Phase 6):** session list with kill button, template editor, audit log viewer, secret rotation trigger.
+- **Stack:** stays unconstrained at the L4 doc level. Likely SPA against the admin REST API. To be designed in a separate `admin-ui.md` once Phase 6 starts (per the per-phase research cadence).
+- **Auth:** OIDC, separate role from end-user.
+
+## Deployment shapes
+
+| Shape | When | Notes |
+|---|---|---|
+| Single binary alongside Compose | PoC, dev (Phase 6 development) | Replaces today's `computer-use-server` container 1:1 |
+| `Deployment` in k8s, HPA on RPS | Production single-region (Phase 6 prod) | Stateless; KV holds session state |
+| Multi-AZ + multi-region | Phase 10 | KV replicated cross-AZ; sticky routing via consistent hashing in L4 (see anti-pattern note below) |
+
+> **Anti-pattern — never use `Service.spec.sessionAffinity: ClientIP`.** It pins on source IP, which in our deployments is the ingress controller's IP (every client looks the same). The result is a single replica receiving 100% of CDP/ttyd WebSocket traffic. Stickiness for long-lived sessions belongs in L4's session router (consistent-hash by `session_id`), not in the Service. Document this in the Helm chart values; refuse the field in any operator template.
+
+## HA upgrade strategy
+
+L4 is stateless at the request level (KV holds session state), but **CDP / ttyd WebSockets are long-lived** and a rolling rollout that kills the replica owning a session terminates that session's UI. The upgrade procedure:
+
+1. **Scale-to-1, migrate, scale-up** is the safe baseline for single-region:
+   - Scale the new revision in.
+   - Drain the old revision: stop accepting new sessions; existing sessions remain bound until they end naturally or until the drain timeout (default 30 min, capped at the longest active CDP session).
+   - Migrate the session router KV (no-op when KV is external; explicit hand-off when KV is co-located).
+   - Scale the old revision to zero.
+2. **Blue-green** (preferred for production):
+   - Stand up the new revision behind a sibling Service.
+   - Switch the L4-fronting ingress route atomically.
+   - Old revision keeps existing sessions; new sessions go to the new revision.
+   - Old revision drains and is deleted after the drain window.
+3. **Never** roll L4 with a vanilla Kubernetes `RollingUpdate` against the same Service while WebSocket sessions are bound. The default kills sessions mid-stream.
+
+Phase 6 ships the scale-to-1 procedure as a Helm `pre-upgrade` hook. Phase 10 adds the blue-green topology once multi-region session affinity is in place.
+
+## Prompt-caching position (Anthropic API gateway role)
+
+L4 sits between LLM clients and the Anthropic API in deployments that route through us. Prompt caching is part of the Anthropic API contract; **L4 must forward the relevant headers and request blocks transparently, not strip or rewrite them**. Specifically:
+
+- The `anthropic-beta: prompt-caching-*` header is set **only** when the client requested it. L4 does not opportunistically add it.
+- `cache_control` blocks inside the request body pass through unchanged.
+- Response headers reporting cache hit/miss metrics pass back unchanged.
+- L4 never inserts or removes `cache_control` markers — that is the client's call. We are a router, not a prompt optimizer.
+
+This matters because Open WebUI and other clients increasingly use caching to amortize long system prompts. A naive proxy that rewrites the body breaks the cache and inflates token bills silently. The L4 implementation has integration tests asserting byte-identical round-trip for cached request shapes.
+
+Phase 6 research locks the exact list of headers and request fields treated as opaque; Phase 6 implementation enforces it.
+
+## Migration from today's FastAPI
+
+- **Phase 1–5:** stay in Python; refactor L3 calls behind a provider interface but `app.py` remains the entrypoint.
+- **Phase 6 (cutover):** new Go service stood up alongside; reverse proxy splits traffic by route; Python service decommissioned once parity is reached and admin UI is migrated.
+- **Compatibility:** Go service MUST accept the exact existing MCP request shape on day 1 — verified by reusing `tests/integration/test_mcp_auth.py` and `test_mcp_tools.py` against the new endpoint.
+
+## Open questions (deferred to Phase 6 research)
+
+- **Web framework:** stdlib `net/http` vs `chi` vs `gin` vs `connect-go` (for gRPC+HTTP unified). Decide in `phase-6-research.md`.
+- **KV choice:** Redis (familiar) vs Valkey (Redis-OSS fork) vs etcd (already in k8s). License sensitivity per [ADR-0006](../adr/0006-no-agpl-no-bsl-dependencies.md).
+- **MCP server library:** roll our own JSON-RPC vs adopt an SDK once mature for Go.
+- **Streaming transport:** SSE vs WebSocket vs HTTP/2 streaming.
+
+These are not blockers to Phases 1–5.

package/docs/future-architecture/architecture/03-layer3-providers.md

@@ -0,0 +1,174 @@
+<!-- SPDX-License-Identifier: FSL-1.1-Apache-2.0 -->
+<!-- Copyright (c) 2025 Open Computer Use Contributors -->
+
+# 03 — Layer 3: Orchestrator / Providers
+
+> Pluggable layer between the control plane (L4) and the sandbox runtime (L2).
+> The interface is **the** key abstraction in this whole roadmap — Phase 1 introduces it.
+
+## The interface
+
+Published as a `.proto` schema from Phase 6 onward (see [ADR-0008](../adr/0008-internal-grpc-external-rest-mcp.md) — connect-go on the wire). Before Phase 6 it lives as a Python `Protocol`, in-process; the shape is identical so the migration is mechanical.
+
+```proto
+service SandboxProvider {
+  rpc Spawn     (SpawnRequest)        returns (SpawnResponse);
+  rpc Configure (ConfigureRequest)    returns (ConfigureResponse);
+  rpc Exec      (ExecRequest)         returns (stream ExecChunk);
+  rpc Upload    (stream UploadChunk)  returns (UploadResponse);
+  rpc Download  (DownloadRequest)     returns (stream DownloadChunk);
+  rpc Stop      (StopRequest)         returns (StopResponse);
+  rpc List      (ListRequest)         returns (stream SandboxHandle);
+  rpc Health    (HealthRequest)       returns (HealthResponse);
+  rpc Events    (EventsRequest)       returns (stream LifecycleEvent);  // provider → control plane
+}
+```
+
+- `SandboxHandle` is opaque to L4; internally it carries provider-specific identifiers (Docker container id, k8s pod ref, VM uuid).
+- `SandboxTemplate` is provider-agnostic — see [09-templates.md](./09-templates.md).
+- `TenantContext` carries `tenant_id`, `session_id`, headers (`X-Chat-Id`, `X-User-Email`), short-lived secrets.
+
+**Transport per phase:**
+- Phase 1 — in-process Python `Protocol`.
+- Phase 2 — HTTP/JSON over the pool-manager sidecar (still Python).
+- Phase 6+ — **connect-go** (gRPC + Connect + HTTP/JSON from one `.proto`). mTLS in production.
+
+The same `.proto` is consumed by L4 (client) and L3 (server). Phase 7 the L1 agent serves a sibling `Agent` service from the same compile, so L3 calls L1 as a typed client.
+
+## Concrete providers
+
+### DockerSocketProvider (PoC, current path)
+
+- **Phase:** in-process today (via `docker_manager.py`); extracted behind the interface in Phase 1; talks HTTP to a pool-manager sidecar from Phase 2.
+- **Backend:** Docker socket — but only this provider knows that. L4 never sees it.
+- **Use:** local dev, single-host PoC, integration tests.
+- **Warm pool:** added in Phase 2 (minSize defaults 0 to preserve current behavior).
+
+### KubernetesProvider
+
+- **Phase:** 5.
+- **Backend:** `kubernetes-asyncio` (Python) or `client-go` (after Phase 6 Go cutover). Talks to k8s API.
+- **CRD basis:** [`kubernetes-sigs/agent-sandbox`](https://github.com/kubernetes-sigs/agent-sandbox) — `Sandbox`, `SandboxTemplate`, `SandboxClaim`, `SandboxWarmPool`. We adopt them rather than inventing our own; we contribute upstream if gaps appear.
+- **Runtime selection:** per-template `runtimeClassName` (runc / sysbox / kata-fc / kata-ch / gvisor).
+- **Network:** default-deny `NetworkPolicy`, egress only via proxy. See [08-networking.md](./08-networking.md).
+- **Replaces today's:** DinD-in-pod pattern. The current Helm chart's inner `docker:dind` sidecar is **transitional only** — Phase 5 replaces it with real per-pod sandboxes.
+
+### DirectCHProvider (optional, Phase 9+)
+
+- **Backend:** drives Cloud Hypervisor directly on a bare-metal host without k8s.
+- **Use:** edge deployments, single-tenant compliance setups, very low-latency requirements.
+- **Trade-off:** no k8s orchestration goodies (HPA, NetworkPolicy, RBAC) — provider implements them itself.
+
+### What we will NOT implement
+
+- **Nomad provider** — Nomad is BSL ([ADR-0006](../adr/0006-no-agpl-no-bsl-dependencies.md)).
+- **Generic OCI provider** — too vague; pick the orchestrator.
+
+## Warm pool semantics
+
+Five knobs per template:
+- `minSize` — sandboxes always idle and ready.
+- `targetSize` — provider tries to maintain (refills as sessions consume from pool).
+- `maxSize` — hard cap regardless of demand.
+- `refillRate` — max sandboxes the refiller may start per second (smooths bursty refill load; without it, a flood of session-ends triggers a thundering-herd spawn).
+- `maxAge` — TTL at which an idle pool sandbox is destroyed and replaced regardless of demand. Prevents long-lived "warm" sandboxes from accumulating per-template image drift, leaked file handles, or stale skill blobs.
+
+Lifecycle:
+1. Provider pre-starts `minSize` sandboxes per template, runs `Configure` with placeholder context.
+2. On `Spawn(template, ctx)` request, pop one from pool, re-`Configure` with real `ctx` (injects session id, env, egress JWT).
+3. Background refill task brings pool back to `targetSize` at no more than `refillRate` per second.
+4. Idle sandbox older than `maxAge` → destroyed, refiller spawns a replacement.
+5. Sessions ending → sandbox is destroyed (not returned to pool — tenancy hygiene; see [07-security.md](./07-security.md)).
+
+Phase 2 ships the skeleton (`minSize=0` default = no behavior change). Phase 5 makes it real. Phase 10 swaps the "warm sandboxes pool" for a **frozen-snapshot pool** with block-device hot-swap on resume — same knobs, different mechanics (internal design note).
+
+## SandboxClaim CRD semantics (KubernetesProvider)
+
+For the `KubernetesProvider`, the wire between L4 and L3 is **typed Kubernetes objects**, not opaque RPC payloads. The CRD shape comes from [`kubernetes-sigs/agent-sandbox`](https://github.com/kubernetes-sigs/agent-sandbox) — we adopt rather than reinvent.
+
+```yaml
+apiVersion: sandbox.kubernetes.io/v1alpha1
+kind: SandboxClaim
+metadata:
+  name: claim-<session-id>
+  namespace: tenant-<tenant-id>
+spec:
+  templateRef:
+    name: customer-cu-kata-ch-v3   # SandboxTemplate to allocate from
+  envtype: managed-hosted           # provider-side dispatch (see below)
+  lease:
+    ttlSeconds: 7200                # auto-release if the session never returns
+    renewDeadlineSeconds: 60        # heartbeat budget
+  context:                          # injected via Agent.Configure
+    sessionId: <id>
+    tenantId:  <id>
+    egressJwtSecretRef:
+      name: egress-token-<session-id>
+status:
+  phase:           Bound | Pending | Released | Failed
+  sandboxRef:      { name, uid }    # opaque to L4; cluster-internal handle
+  boundAt:         <timestamp>
+  observedRuntime: kata-ch          # actual L2 runtime that landed
+  conditions:     [...]
+```
+
+L4's `Spawn` becomes "create a `SandboxClaim`, watch `.status.phase`." `Stop` is `delete claim`. Health is the same watch. The CRD is what gives the provider a place to store **lease state** without L4 caring about Kubernetes specifics.
+
+Two operational properties we get for free:
+- **TTL-driven cleanup.** Claims past their lease are released by the controller, not by L4. L4 crashing does not strand sandboxes.
+- **kubectl-debuggable.** Operators can `kubectl get sandboxclaims -A` to see pool state without going through L4's admin API.
+
+The `DockerSocketProvider` and `DirectCHProvider` implement the same lifecycle in-process; the CRD shape is the k8s realization of a provider-internal concept.
+
+## Environment-type dispatch
+
+Templates carry an `envtype` field consumed by the provider to pick the backend mechanism. The pattern follows an industry-observed router/session-agent split and is applied narrowly here:
+
+| `envtype` | Provider behaviour | Use case |
+|---|---|---|
+| `dev` | runc on Docker Compose; no isolation; no egress proxy | Local PoC, integration tests |
+| `internal` | sysbox on k8s; egress proxy in monitor mode | Trusted employees |
+| `customer-shared` | sysbox or gVisor (per-template) on k8s; egress proxy enforcing | Customer code-only sandboxes |
+| `customer-cu` | Kata (CH or FC) on bare-metal node pool; egress proxy enforcing | Customer Computer Use sessions |
+| `managed-hosted` | Reserved label for our own managed deployment; same as `customer-cu` today but pinned to a tier | Managed-deployment shape |
+| `byoc` | Customer-supplied cluster; provider holds a lease on a customer namespace | Reserved, not Phase-1 |
+
+`envtype` is **not** the same as `runtimeClass`. `runtimeClass` is the L2 isolation primitive; `envtype` is the L3 dispatch key. One `envtype` can map to multiple `runtimeClass`-es depending on template (e.g. `customer-shared` resolves to sysbox for code, gVisor for browserless code-exec).
+
+## Reaper / cleanup
+
+- Current implementation: per-container Python thread + cron sidecar.
+- Target: provider-owned background task, idle-timeout per template, reaped synchronously when stopping. Cron sidecar is removed.
+
+## Tenancy & isolation
+
+- Per `tenant_id`: dedicated k8s namespace (`KubernetesProvider`) or dedicated network (`DockerSocketProvider`).
+- `NetworkPolicy`: deny workspace→workspace, deny workspace→control-plane (except via the egress proxy and the L4-managed exec path).
+- `ResourceQuota` + `LimitRange` per namespace (k8s only) — blast-radius containment.
+- Per-sandbox `ServiceAccount` with **empty** RBAC (no cluster enumeration possible).
+
+## Events
+
+Provider emits structured events the control plane consumes for audit + UI:
+- `sandbox.spawned`, `sandbox.configured`, `sandbox.exec.started/completed`, `sandbox.stopped`, `sandbox.health.degraded`, `sandbox.evicted`.
+
+Transport: same channel as L4 ↔ L3 (HTTP stream or gRPC server-side stream).
+
+## Phase-by-phase progression for L3
+
+| Phase | What changes |
+|---|---|
+| 1 | Interface extracted; `DockerSocketProvider` is the only impl; still in-process |
+| 2 | HTTP transport; pool-manager sidecar owns Docker socket; warm pool skeleton |
+| 3 | Storage (S3) plumbed via provider (mount specs in template) |
+| 4 | Secret broker integration — provider receives short-lived creds in `configure` |
+| 5 | `KubernetesProvider` ships; Helm chart real per-pod sandboxes |
+| 7 | Provider learns to pass new Go-agent endpoints (vsock-ready spec) |
+| 9 | `DirectCHProvider` ships; templates gain `runtimeClass` selection |
+| 10 | Snapshot / restore + multi-region session routing |
+
+## Source
+
+- Internal design notes (Layer 3 sections)
+- [`docs/future-architecture/architecture/01-layers.md`](./01-layers.md)
+- [`docs/future-architecture/references.md`](../references.md) (`kubernetes-sigs/agent-sandbox`, `e2b-dev/infra`)

package/docs/future-architecture/architecture/04-layer2-runtimes.md

@@ -0,0 +1,114 @@
+<!-- SPDX-License-Identifier: FSL-1.1-Apache-2.0 -->
+<!-- Copyright (c) 2025 Open Computer Use Contributors -->
+
+# 04 — Layer 2: Sandbox Runtimes
+
+> The actual isolation primitive that wraps Layer 1 (the agent + workload).
+> Selected **per template**, not globally. Same agent, same orchestrator, different runtime → different threat-model fit.
+
+## Runtime matrix
+
+| Runtime | Cold start | RAM overhead | Isolation | Use case | Status in our stack |
+|---|---|---|---|---|---|
+| **runc** | ~30 ms | ~2 MB | Namespaces only — none for untrusted code | Dev / CI | Today via Docker Compose |
+| **sysbox** | ~50 ms | ~5 MB | + user-ns remap, procfs emulation; shares host kernel | Internal trusted users; DinD; systemd | Today via Helm (current default) |
+| **gVisor** | ~50 ms | ~15-25 MB | Userspace kernel intercepts syscalls | Code-execution sandboxes (no browser) | Experimental, Phase 7+ |
+| **Kata + Firecracker** (`kata-fc`) | ~125 ms | ~5-10 MB | KVM hypervisor, minimal device model | Public untrusted, fastest cold start | Phase 9 |
+| **Kata + Cloud Hypervisor** (`kata-ch`) | ~150 ms | ~10-20 MB | KVM hypervisor + virtio-fs + GPU passthrough | **Computer Use (browser)** untrusted | Phase 9 (primary target) |
+| **Kata + QEMU** | ~500 ms | ~50 MB | KVM hypervisor, full device model | Compatibility fallback | Not planned |
+
+Numbers from internal design notes. Validate during Phase 9 research on actual hardware.
+
+## Why Cloud Hypervisor is the lead untrusted runtime (not Firecracker)
+
+- **virtio-fs** — fast shared mounts; Firecracker omits this. Important for skill blobs and user-data overlays (see [06-storage.md](./06-storage.md)).
+- **GPU passthrough** — relevant if Computer Use workloads ever need accelerated rendering.
+- **Hot-plug** — easier resource adjustments.
+- Trade-off: ~80K LoC vs Firecracker's ~50K (larger attack surface, still small).
+
+Firecracker stays available via `kata-fc` for the fastest-cold-start tier (e.g., free-tier anonymous trials). Note that Firecracker is the microVM that AWS Lambda and Fargate are built on — its scale-pattern lineage informs the Phase 10 snapshot-pool design without making us a Lambda deployment. See the Lambda framing in [`references.md`](../references.md) and [ADR-0010](../adr/0010-lambda-as-inspiration-not-runtime.md).
+
+## virtio-fs vs 9p — the CH/FC asymmetry
+
+CH and FC do not agree on shared-filesystem story, and the difference is load-bearing for Tier-2 (skills) and Tier-4 (user data) mounts:
+
+| | virtio-fs | 9p |
+|---|---|---|
+| Cloud Hypervisor | First-class (default) | Possible but not the natural path |
+| Firecracker upstream | **Not supported** in stock Firecracker | The historical option; out-of-tree patches and Kata wrappers exist |
+| Performance | Native-ish (FUSE protocol, shared page cache) | Slower; protocol overhead dominates |
+| Posix coverage | High | Lower (the legacy choice) |
+
+Implication: `kata-ch` is the only tier where Tier-2 / Tier-4 mounts are "free." On `kata-fc` we either accept 9p's performance/POSIX trade-offs, lean on a FUSE client inside the VM, or block-device-mount squashfs. Phase 9 research locks the choice per tier.
+
+## nydus snapshotter for lazy image-layer load
+
+For the microVM tiers (`kata-fc`, `kata-ch`), pulling the full container image at sandbox spawn is the single biggest cold-start cost. **nydus** ([nydus-snapshotter](https://github.com/containerd/nydus-snapshotter), Apache 2.0) reformats OCI images into a chunk-addressable layout that the VM can lazy-load on demand — pages are fetched as files are touched, not upfront.
+
+- **Relevance.** Phase 9 cold-start budget for `kata-ch` is in the 100–200 ms range with full image pull. Lazy-load with nydus gets that closer to template-snapshot territory without the snapshot-pool engineering bill.
+- **Trade-off.** Adds a new component to the runtime path; failure modes (registry hiccups mid-execution) need their own playbook.
+- **Decision.** Phase 9 research evaluates whether nydus or a snapshot pool (or both, layered) hits the cold-start target.
+
+## VMM Lambda lineage (one paragraph, by reference)
+
+Firecracker exists because AWS needed a VMM small enough to scale Lambda/Fargate. Our Phase-9 `kata-fc` tier inherits from that lineage. The architectural takeaway is the **VMM design** (minimal device model, small attack surface, fast init) — not the deployment substrate. See [`references.md`](../references.md) Lambda framing, [`research/05`](../research/05-firecracker.md), and [ADR-0010](../adr/0010-lambda-as-inspiration-not-runtime.md) for the closed answer to "are we going to run on Lambda?" (no).
+
+## Why NOT gVisor for browsers
+
+Already a locked decision from the pre-existing `docs/requirements/k8s-architecture.md`:
+
+> compatibility envelope too narrow for Chromium with sandbox flags, Playwright, browser downloads
+
+gVisor remains viable for non-browser code-execution sandboxes (e.g., a "run this Python snippet" tier). Phase 7 validates this as an optional experimental tier.
+
+## Selection mechanism
+
+- **In k8s:** `Pod.spec.runtimeClassName` — installed via `kata-deploy` DaemonSet (for kata-*) and `gvisor` runtimeclass.
+- **Direct (DirectCHProvider):** hypervisor invocation, no k8s.
+- **In Docker Compose:** runc only (the PoC); sysbox optional if the host has it. Compose is not the prod runtime story.
+
+`SandboxTemplate.runtime_class` carries the choice; the provider plumbs it. See [09-templates.md](./09-templates.md).
+
+## Threat-model matrix (target tiering)
+
+| Tenant tier | Workload | Runtime |
+|---|---|---|
+| Internal employees + trusted scripts | Code only | sysbox (or runc in dev) |
+| Internal employees + Computer Use | Browser, file ops | sysbox |
+| External customer + code only | Code only | gVisor or `kata-ch` |
+| External customer + Computer Use | Browser | `kata-ch` |
+| Anonymous trial | Anything | `kata-fc` |
+
+The control plane (L4) picks the template (and thereby the runtime) based on tenant tier at session-spawn time.
+
+## Hardware / cluster requirements
+
+- **runc, sysbox, gVisor:** any Linux kernel ≥ 5.x. Run on any cloud VM, including managed k8s (EKS, GKE on standard nodes).
+- **kata-fc, kata-ch:** require **bare-metal** k8s nodes — KVM is needed and nested-virt won't reliably work in most cloud VMs.
+  - On AWS: `m6i.metal` / `c6i.metal` etc.
+  - On-prem RKE2: any host with `/dev/kvm`.
+  - Use a **dedicated node pool with taints** to keep regular workloads off the bare-metal nodes (they're expensive).
+
+## What ships, when
+
+- **Phase 1–4:** runc only (via Docker Compose) and sysbox (via existing Helm chart). No L2 change.
+- **Phase 5:** real `KubernetesProvider` ships with sysbox as default; the Helm chart switches from DinD-in-pod to real per-pod sandboxes on sysbox.
+- **Phase 7:** gVisor added as experimental tier for non-browser sandboxes; runtime selection becomes per-template.
+- **Phase 9:** `kata-ch` and `kata-fc` added; bare-metal node pool required; multi-tier templates land.
+
+## Security boundary per runtime (one-liner)
+
+| Runtime | Primary boundary | Boundary fails if… |
+|---|---|---|
+| runc | Linux namespaces | …kernel CVE (Dirty Pipe, nf_tables) — assume escapable |
+| sysbox | Above + user-ns + emulation | …kernel CVE; sysbox bugs |
+| gVisor | Sentry userspace kernel (Go) | …Sentry bug; passthrough syscall path |
+| kata-fc | KVM + Firecracker VMM | …Firecracker CVE; KVM CVE; side-channel |
+| kata-ch | KVM + Cloud Hypervisor VMM | …CH CVE; KVM CVE; side-channel |
+
+See [07-security.md](./07-security.md) for the full threat model.
+
+## Source
+
+- Internal security and architecture notes
+- [`docs/future-architecture/references.md`](../references.md) (every runtime URL listed there)