@mseep/open-computer-use 1.0.0

package/docs/future-architecture/research/01-kata-containers.md

@@ -0,0 +1,100 @@
+<!-- SPDX-License-Identifier: FSL-1.1-Apache-2.0 -->
+<!-- Copyright (c) 2025 Open Computer Use Contributors -->
+
+# 01 — Kata Containers (Rust agent + kata-deploy)
+
+> Source: [kata-containers/kata-containers](https://github.com/kata-containers/kata-containers) (Rust PID 1 agent + k8s DaemonSet for installing Kata on nodes).
+> Relevant to Phase 7 (Go guest agent — port these patterns) and Phase 9 (Kata + Cloud Hypervisor for untrusted tier).
+
+## 1. PID 1 — Subreaper + async `SIGCHLD` loop
+
+- **What it does.** Register as subreaper via `prctl(PR_SET_SUBREAPER, 1)`, then await `SIGCHLD` in a tokio loop; on each signal call `waitpid(-1, WNOHANG | __WALL)` under a lock to reap orphans while keeping a sandbox-owned process map consistent.
+- **Where.** `src/agent/src/signal.rs:9-122` (full file). `set_subreaper(true)` at line 95; `handle_sigchild()` loop at lines 21–86; signal setup at 88–122.
+- **Port to Go.**
+  1. `unix.Prctl(unix.PR_SET_SUBREAPER, 1, 0, 0, 0)` at startup.
+  2. `signal.Notify(ch, syscall.SIGCHLD)`.
+  3. Loop with `syscall.Wait4(-1, &wstatus, syscall.WNOHANG|syscall.WALL, nil)`.
+  4. Track exits in the agent's process map.
+- **Skip.** Kata's `WAIT_PID_LOCKER` is container-runtime specific; our product lifecycle differs.
+
+## 2. vsock listener — `AddressFamily::Vsock` + stream binding
+
+- **What it does.** Bind `AF_VSOCK` on port N, listen, accept. Used for debug console and log streaming without unix sockets.
+- **Where.** `src/agent/src/main.rs:161-183` (`create_logger_task`). Socket at line 165; bind+listen at 171–173; `VsockAddr::new(VMADDR_CID_ANY, vsock_port)` at 171.
+- **Port to Go.**
+  1. `github.com/mdlayher/vsock`: `vsock.Listen(":2048")`.
+  2. Accept and demultiplex by service (health, API, logging) on connection.
+  3. Graceful fallback if vsock unavailable (matches our cross-cutting pattern 3).
+- **Skip.** Kata's ttrpc protocol is OCI-shaped.
+
+## 3. gRPC service definitions — structure to learn, NOT copy
+
+- **Where.** `src/libs/protocols/protos/agent.proto:20-82`. 40+ RPCs: container lifecycle (`CreateContainer`/`StartContainer`/`RemoveContainer`, 22–32), process control (`ExecProcess`/`SignalProcess`/`WaitProcess`, 33–35), stdio multiplexing (`WriteStdin`/`ReadStdout`/`ReadStderr`, 44–49), networking (51–58), `GetMetrics` (61).
+- **Take.** Lifecycle-phase separation; stdio model (request/response for stdin, server-push events for stdout/err); device-hotplug semantics post-VM-start.
+- **Skip.** The OCI shape itself — our agent API is **product-aware**, not generic OCI ([`architecture/05-layer1-guest-agent.md`](../architecture/05-layer1-guest-agent.md)).
+
+## 4. Hardening at startup — init-as-PID-1 setup
+
+- **What it does.** When PID 1: mount cgroups v1/v2, set hostname, `setsid()`, set controlling terminal via `ioctl`, configure `PATH`.
+- **Where.** `src/agent/src/main.rs:648-680` (`init_agent_as_init`). Cgroup mount: 651. `/dev/ptmx` symlink: 659–660. `setsid()`: 662. Controlling-tty ioctl: 665. Hostname: 670–677.
+- **Port to Go.** Detect `getpid() == 1`; conditionally run init routine; mount cgroups only if absent; symlink `/dev/ptmx` if missing.
+- **Skip.** Full OCI init (hooks, env setup) — ours is microVM-specific, much smaller.
+- **Note.** `PR_SET_DUMPABLE=0` and capability drops live in the **runtime config**, not the agent (agent runs as root inside the guest). We pair this with our cross-cutting pattern 5.
+
+## 5. kata-deploy DaemonSet — install/cleanup, probes, node affinity
+
+- **What it does.** Per-node DaemonSet copies Kata binaries, configures CRI (containerd/CRI-O), creates `RuntimeClass` resources, cleans up on terminate. Node affinity filters on CPU virt features (VMX/SVM).
+- **Where.** `tools/packaging/kata-deploy/helm-chart/kata-deploy/templates/kata-deploy.yaml:1-384`.
+  - DaemonSet: 21–38.
+  - Virt-affinity (x86 VMX/SVM): 77–130.
+  - Install action: 140.
+  - Probes (startup/liveness/readiness): 317–344.
+  - hostPath mounts: 349–379.
+  - `terminationGracePeriodSeconds: 600`: line 135.
+- **Take for Phase 9 Helm work.**
+  - `hostPID: true` for in-container runtime restart visibility.
+  - Generous `terminationGracePeriodSeconds` for cleanup.
+  - Startup probe with many short retries (60×10 s = 600 s budget).
+  - Affinity on hardware capability (KVM-capable nodes only).
+  - Env-driven per-node config (shim selection, etc.).
+- **RuntimeClass setup.** `tools/packaging/kata-deploy/binary/src/k8s/runtimeclasses.rs:11-87` — list existing `kata-*` classes, patch `overhead.podFixed` from NFD labels (e.g. `tdx.intel.com/keys`, `sev-snp.amd.com/esids`).
+- **Skip.** Multi-install suffix (parallel Kata versions), NFD complexity, multi-arch shim selection.
+
+## 6. Configuration — TOML structure per hypervisor backend
+
+- **What it does.** Runtime config split per backend: `[hypervisor.qemu]`, `[hypervisor.firecracker]`, `[hypervisor.clh]`. Host selects active backend by hardware/policy.
+- **Where.** `src/runtime/config/configuration-clh.toml.in:14-28`, `configuration-fc.toml.in:14-40`, `configuration-qemu.toml.in:14-80`.
+- **Common knobs per backend.** `path` (hypervisor binary), `kernel`, `image` (guest rootfs), `rootfs_type` (ext4/xfs/erofs), `default_vcpus`, `kernel_params`, annotation allowlists.
+- **Take for Phase 9 host shim.**
+  1. One section per backend in host config.
+  2. Defaults per backend (vCPUs, memory overhead, kernel params differ).
+  3. Annotation allowlists for which fields pod-author can override.
+- **For the Go agent.** Agent doesn't parse this — host passes choices via `/proc/cmdline`. Agent extracts backend identity to decide feature set (e.g., TEE only on CH).
+
+## 7. Backend switching — Cloud Hypervisor vs Firecracker vs QEMU
+
+- **Where.** `src/runtime/config/` (13 backend configs total). FC has `jailer_path`; CH has `firmware` for TEEs.
+- **For the agent.** Detect backend at startup via kernel cmdline (`kata.hypervisor=clh`) or DMI/CPUID markers; toggle feature flags (TEE attestation enabled only under CH).
+
+## 8. Small-binary Rust patterns — applicable to Go too
+
+- **Where.** `src/agent/Cargo.toml:1-109`. Workspace deps (105–109), `profile.release` LTO (102–103).
+- **Result.** ~3–5 MB unstripped, ~1.5 MB stripped.
+- **Port to Go.**
+  - Build tags for optional services (policy, confidential data hub).
+  - `CGO_ENABLED=0 go build -ldflags="-s -w -X main.version=$VERSION"`.
+  - Expected: ~5–10 MB for a production Go agent — acceptable.
+  - UPX worth testing but Kata doesn't use it.
+
+## Adoption matrix
+
+| Pattern | Adopt? | Why |
+|---|---|---|
+| PID 1 subreaper + `SIGCHLD` loop | YES | Mandatory for PID 1 |
+| vsock listener on fixed port | YES | Standard microVM transport |
+| OCI ttrpc service structure | NO (study) | Our API is product-shaped |
+| init-as-PID-1 setup | PARTIAL | Adapt to minimal microVM init |
+| DaemonSet + probes + node affinity | YES | Phase-8 Helm pattern |
+| TOML config per backend | YES | Host shim, not agent |
+| Runtime-backend detection | YES | Feature toggling |
+| Feature flags + LTO / `-s -w` | YES | Small Go binary |

package/docs/future-architecture/research/02-e2b-infra.md

@@ -0,0 +1,133 @@
+<!-- SPDX-License-Identifier: FSL-1.1-Apache-2.0 -->
+<!-- Copyright (c) 2025 Open Computer Use Contributors -->
+
+# 02 — E2B Infrastructure (e2b-dev/infra)
+
+> Source: [e2b-dev/infra](https://github.com/e2b-dev/infra). Production E2B infra in Go.
+> Most relevant for Phase 2 (HTTP pool sidecar), Phase 3 (storage), Phase 6 (control plane), Phase 7 (guest agent), Phase 8 (egress proxy).
+
+## 1. Network slot pool — dual-channel recycle with drain delay
+
+- **What.** Pool of network-namespace slots with **two** channels — "new" (pre-allocated at startup) and "reused" (returned post-drain). `Get()` tries reused first, falls back to new. `Return()` waits `ReturnDelay` (3 s) before pushing back to allow inflight requests to drain.
+- **Where.** `packages/orchestrator/pkg/sandbox/network/pool.go` (357 lines).
+  - `NewPool()` builds dual channels.
+  - `Get()` tries reused, falls back to new.
+  - `Return()` schedules recycle with `ReturnDelay`.
+  - Metrics: `newSlotsAvailableCounter`, `reusableSlotsAvailableCounter`.
+- **Why for us.** Phase 2 pool semantics. The drain delay prevents dropped connections when sandboxes cycle fast.
+- **Skip.** Linux iptables/netlink specifics; keep the slot/return semantics as an abstraction.
+
+## 2. Adjustable semaphore — separates fresh-create vs snapshot-resume
+
+- **What.** Feature-flag-driven semaphore limits concurrent **sandbox starts**. Snapshot resumes use `waitForAcquire` (15 s timeout — higher parallelism). Fresh creates use `TryAcquire` (immediate or reject).
+- **Where.** `packages/orchestrator/pkg/server/main.go:87-91, 162-182` (`NewAdjustableSemaphore`, `refreshStartingSandboxesLimit` every 30 s). Usage: `packages/orchestrator/pkg/server/sandboxes.go:116-130`.
+- **Why for us.** Phases 2 + 6. Prevents thundering herd on template loads / NBD allocation / memory.
+- **Skip.** LaunchDarkly — use simple env / config.
+
+## 3. envd — PID 1 Go agent with Connect-RPC streaming exec
+
+- **What.** PID 1 in each sandbox; `os/exec` + signal forwarding; Connect-RPC over HTTP/2 in a single binary; `Process.Start()` streams stdout/stderr/pty as a oneof event; supports `SendSignal` (SIGTERM/SIGKILL), stdin/pty write, KeepAlive events for idle-TCP survival.
+- **Where.**
+  - Main: `packages/envd/main.go:132-221` (HTTP/2 server, chi router, Connect auth).
+  - Service: `packages/envd/internal/services/process/service.go:19-84`.
+  - Handler: `packages/envd/internal/services/process/handler/handler.go:44-487`.
+  - Proto: `packages/envd/spec/process/process.proto:1-172`.
+- **Why for us.** **Direct template for Phase 7 Go agent.** Pattern set:
+  - Signal forwarding via SDK call.
+  - Streaming output via oneof event (stdout / stderr / pty / keepalive / exit).
+  - Multiplex multiple concurrent execs per VM.
+  - KeepAlive frames against TCP idle timeout.
+- **Skip.** Firecracker MMDS polling, vsock specifics, cgroup v2 (port the *shape*, not Linux glue).
+
+## 4. Sandbox creation flow — multi-resource assembly with rollback
+
+- **What.** `SandboxCreateRequest` acquires in sequence: network slot, template, NBD block device, memory+rootfs snapshots. Feature flags gate Firecracker version, max sandboxes/node, internet access, disk size. Returns `client_id` for session routing.
+- **Where.** `packages/orchestrator/pkg/server/sandboxes.go:60-235`.
+  - 107–129: semaphore + node capacity.
+  - 132–141: template fetch.
+  - 143–161: network/egress config.
+  - 163–214: assemble + `ResumeSandbox()`.
+  - 215–235: rollback on failure.
+  - 237–249: lifecycle hooks + event publish.
+- **Why for us.** Phase 6 orchestration. The **rollback pattern** (release acquired resources on partial failure) is the key takeaway.
+- **Skip.** Nomad job scheduling, GCS template bucket. Keep the multi-resource orchestration shape.
+
+## 5. Cgroup v2 isolation — weighted process classes via `clone3(CLONE_INTO_CGROUP)`
+
+- **What.** Three cgroup hierarchies per sandbox:
+  - **PTYs (interactive):** `cpu.weight=200`, `memory.high=80%`.
+  - **Socats (proxies):** `cpu.weight=150`, `memory.min/low=8MB`.
+  - **User processes:** `cpu.weight=50`, `memory.high=80%`.
+  Uses `clone3(CLONE_INTO_CGROUP)` with a passed file descriptor for **race-free** classification at process birth.
+- **Where.** Manager `packages/orchestrator/pkg/sandbox/nbd/cgroup/cgroup2.go:1-120`. envd integration: `packages/envd/main.go:223-272`.
+- **Why for us.** Phase 7 — prevents user processes from starving orchestrator infrastructure inside the sandbox. The Firecracker memory limit is whole-VM; this gives per-process guarantees.
+
+## 6. Egress proxy — protocol-specific inspection ports
+
+- **What.** Single `tcpproxy.Proxy` listening on three ports:
+  - **HTTP (5016):** inspects `Host` header against allowlist.
+  - **TLS (5017):** inspects SNI against allowlist.
+  - **Other (5018):** CIDR-only check (no protocol sniffing — prevents blocking SSH).
+  Host iptables redirects by original dst port.
+- **Where.** `packages/orchestrator/pkg/tcpfirewall/proxy.go:1-100+`.
+- **Why for us.** Phase 8 — domain blocklist without false positives from protocol mis-detection.
+- **Skip.** iptables/netlink Linux specifics; abstract as "traffic shaper with protocol inspection". Compare with our planned [agentbox-style](./09-agentbox.md) JWT pattern — these are complementary (this filters; JWT authorizes).
+
+## 7. Template streaming cache — lazy block-device loading
+
+- **What.** Snapshots (memfile, rootfs, metadata) keyed by `buildID`. First access streams from GCS/S3 into local tmpfs; cached for 1 h. Supports layered builds.
+- **Where.** Interface `packages/orchestrator/pkg/sandbox/template/template.go:16-24`. Proto `packages/orchestrator/template-manager.proto:1-179` (layer upload at 9–18, config at 61–89, metadata at 127–135).
+- **Why for us.** Phase 3 — templates as **streaming block devices**, not OCI images. Layer-reuse via `cacheScope`.
+
+## 8. Slot recycling — graceful return with locking discipline
+
+- **What.** On sandbox end, slot recycle waits `ReturnDelay` for drain → resets internet config (iptables) → returns to pool. RWMutex on the reused-slots channel; lock released **before** slow `RemoveNetwork` syscalls (lines 254–286 of `pool.go`).
+- **Where.** `packages/orchestrator/pkg/sandbox/network/pool.go:204-298`. Metrics: `returnedSlotCounter`, `releasedSlotCounter`.
+- **Why for us.** Phase 2. Same drain + lock discipline ensures fast recycle without dropping inflight requests or stalling new acquisitions.
+
+## 9. Per-sandbox metrics — delta-temporality observable gauges
+
+- **What.** Callback-driven observable gauges (CPU %, mem bytes, disk bytes) measured in parallel (5× concurrency cap). Uses OTel **delta temporality** so gauges don't repeat indefinitely after sandbox death. Tagged by `sandbox_id`, `team_id`, `build_id`. Warns at >80 % mem or CPU.
+- **Where.** `packages/orchestrator/pkg/metrics/sandboxes.go:46-319`. Export every 5 s (line 41).
+- **Why for us.** Phase 6 / 10. Especially the **delta temporality** detail — common foot-gun with OTel gauges in ephemeral-pod environments.
+
+## 10. API gateway auth — multi-tenant team-context extraction
+
+- **What.** Request validated by either API token (Bearer → team lookup) or Supabase JWT (user → teams → default team). All downstream calls receive `teamID` context.
+- **Where.** `packages/api/internal/handlers/auth.go:1-80`.
+- **Why for us.** Phase 6. Foundational multi-tenant pattern; same shape regardless of identity backend.
+- **Skip.** Supabase specifics; substitute our OIDC provider.
+
+## 11. Snapshot/pause/resume lifecycle
+
+- **What.** On pause: save VM memory + disk to GCS/S3. On resume: reconstruct from snapshot files; `Server.Create()` branches on `req.Sandbox.Snapshot` (line 69, 117) — resume uses `resumeVM()` with `waitForAcquire(15s)` vs. fresh `TryAcquire`.
+- **Where.** Create flow: `packages/orchestrator/pkg/server/sandboxes.go:69, 117-129`. FC client: `packages/orchestrator/pkg/sandbox/fc/client.go` (`resumeVM`, `pauseVM`).
+- **Why for us.** Phase 10. Stateful handoff, cost optimization, multi-session continuity.
+
+## 12. Version-gated metric collection
+
+- **What.** Observe metrics only if `envd.Version >= minEnvdVersionForMetrics` (line 165). Feature-specific minima: memory precision ≥ 0.2.4, disk ≥ 0.2.4, cache ≥ 0.5.9. Prevents crashes on older sandboxes.
+- **Where.** `packages/orchestrator/pkg/metrics/sandboxes.go:165-173, 225-257`.
+- **Why for us.** Phase 6 — safe gradual rollout of new metrics without coordinating sandbox upgrades. General Go pattern, adopt broadly.
+
+## 13. Zombie reaping via Go stdlib `os/exec.Wait()`
+
+- **What.** Per spawned process: goroutines stream stdout/stderr/pty; `Process.Wait()` blocks on `cmd.Wait()`; SIGTERM/SIGKILL cancels output context (`p.outCancel()` line 360).
+- **Where.** `packages/envd/internal/services/process/handler/handler.go:335-486`. Start: 429–447. Wait: 449–486 (cmd.Wait at 453, exit at 464–469). Signal: 354–364.
+- **Why for us.** Phase 7. Go stdlib handles `SIGCHLD` automatically for *its own* children — but for PID 1 reaping of inherited orphans we still need [pattern 1 from kata](./01-kata-containers.md#1-pid-1--subreaper--async-sigchld-loop). Use both.
+
+## Summary
+
+| Pattern | File | Phase | Take? |
+|---|---|---|---|
+| Dual-channel network pool + drain delay | `orchestrator/pkg/sandbox/network/pool.go` | 2 | YES |
+| Adjustable semaphore (fresh vs resume) | `orchestrator/pkg/server/main.go` | 2,6 | YES |
+| envd Connect-RPC streaming agent | `envd/main.go`, `envd/internal/services/process/` | 7 | YES — template |
+| Multi-resource create with rollback | `orchestrator/pkg/server/sandboxes.go` | 6 | YES |
+| Cgroup v2 weighted classes | `orchestrator/pkg/sandbox/nbd/cgroup/cgroup2.go` | 7 | YES |
+| Three-port egress firewall | `orchestrator/pkg/tcpfirewall/proxy.go` | 9 | YES |
+| Streaming template cache | `orchestrator/pkg/sandbox/template/` | 3 | YES |
+| Delta-temporality observable gauges | `orchestrator/pkg/metrics/sandboxes.go` | 6,10 | YES |
+| Multi-tenant team-context auth | `api/internal/handlers/auth.go` | 6 | YES |
+| Snapshot pause/resume | `orchestrator/pkg/server/sandboxes.go` | 10 | YES |
+| Version-gated metric features | `orchestrator/pkg/metrics/sandboxes.go` | 6 | YES |

package/docs/future-architecture/research/03-coder.md

@@ -0,0 +1,115 @@
+<!-- SPDX-License-Identifier: FSL-1.1-Apache-2.0 -->
+<!-- Copyright (c) 2025 Open Computer Use Contributors -->
+
+# 03 — Coder (production Go control plane)
+
+> Source: [coder/coder](https://github.com/coder/coder). Self-hosted workspace platform — auth, sessions, lifecycle, templates, audit, RBAC, telemetry. Closest production analog for our Phase 6 L4 rewrite.
+> Analysis covers AGPL OSS edition (May 2026).
+
+## 1. HTTP bootstrap — Chi router with layered middleware
+
+- **Where.** `coderd/coderd.go:511-1050`.
+- **What.** `go-chi/chi/v5` router with deliberate middleware ordering: recovery → request ID (`httpmw.AttachRequestID`) → real-IP → Prometheus → route match → rate limit → CORS → API key extraction.
+- **Why for us.** Phase 6. We'll hit 150+ endpoints; Chi's `.Route()` groups + middleware composition scale. Ordering matters — observability before route match so metrics include unmatched paths; auth populates context for later handlers.
+- **Skip.** Their swaggo integration and cookie-CSRF (specific to their cookie-auth scheme).
+
+## 2. Auth — dual-mode API key, JWT signing keys rotated in DB
+
+- **Where.** `coderd/httpmw/apikey.go:42-100` (validation config) and `:100-500+` (extraction + role loading). OAuth state JWT: `coderd/userauth.go:50-200`.
+- **What.** `PrecheckAPIKey` middleware validates early; `ExtractAPIKey` loads session lazily. Tokens in cookie (browser) or `Authorization` header. **No Redis** — sessions are DB rows. Signing keys for OAuth state JWTs are managed in DB by `cryptokeys.StartRotator` (`coderd/coderd.go:588`) — rotation is centralized and zero-touch.
+- **Why for us.** Avoids the session-store sync problem in HA L4 deployments. Key rotation pattern directly applicable to our secret broker ([architecture/07-security.md](../architecture/07-security.md)).
+- **Skip.** OAuth-provider mode (they act as both consumer and provider), multi-IDP sync. Phase-6 MVP needs only consumer-mode OIDC.
+
+## 3. RBAC — Rego policy + DB role reconciliation + `dbauthz` wrapper
+
+- **Where.**
+  - Policy: `coderd/rbac/policy.rego`.
+  - Auth check: `coderd/rbac/authz.go:1-100`.
+  - Authorizer creation: `coderd/coderd.go:361-365`.
+  - Reconciler: `coderd/rbac/rolestore/reconcile.go` (startup-time DB ↔ code sync).
+  - DB enforcement: `coderd/database/dbauthz/` wraps every query.
+- **Why for us.** Phase 6. Three takeaways:
+  1. Policy-as-code in Rego — auditable, testable in isolation.
+  2. **Reconcile system roles at startup** — guarantees code/DB consistency.
+  3. **`dbauthz` wrapper** — single enforcement point; "you cannot query the DB without auth context". Strong default against accidental data leaks.
+- **Skip.** Workspace-sharing ACLs (enterprise), full Rego policy initially (start simple, grow into it). OAuth2 scopes (only if we become a provider).
+
+## 4. Audit log — DB-backed with field-level sensitivity catalog
+
+- **Where.** `coderd/audit.go:1-100`. Field catalog: `enterprise/audit/table.go`. Queries: `coderd/database/queries/auditlogs.sql`.
+- **What.** `audit_logs` rows carry structured JSON diffs (before/after) per field. Each field declared `ActionTrack` | `ActionIgnore` | `ActionSecret` — secrets never serialized.
+- **Why for us.** Phase 6 + 9. Same DB as system → no sync issues. Field-level sensitivity = mechanical enforcement of our cross-cutting pattern 10 (never log verbatim).
+- **Skip.** Real-time export to Splunk/DataDog (Phase 8 may add via separate sink), enterprise retention policies. Start DB-only.
+
+## 5. Workspace lifecycle — provisioner job abstraction
+
+- **Where.** `coderd/workspacebuilds.go:1-100`, `coderd/workspaces.go:897-984`, queries `coderd/database/queries/workspacebuilds.sql`, status: `coderd/provisionerjobs.go`.
+- **What.** Workspace create/update/delete = a **provisioner job** row (status: pending/running/succeeded/failed). Watchable over WebSocket; logs + result captured. Actual work delegated to separate provisioner daemons via gRPC.
+- **Why for us.** Phase 6. Same shape works for our sandbox spawn/destroy: observable, retryable, decoupled from the L3 provider implementation. Logs persisted = "why did spawn fail?" stays answerable.
+- **Skip.** Pre-build caching, Terraform variable interpolation, port-forwarding/SSH (workspace-specific).
+
+## 6. Templates — `Template` + `TemplateVersion` + `ParameterSchema`
+
+- **Where.** `coderd/templateversions.go:1-150`, `coderd/templates.go:1-100`, schemas in `coderd/database/queries/`.
+- **What.** `Template` = immutable metadata. `TemplateVersion` = versioned artifact (source + params + timestamp). `ParameterSchema` = validated HCL inputs at creation time.
+- **Why for us.** Phase 6, validates our [`09-templates.md`](../architecture/09-templates.md) `SandboxTemplate` versioning approach. Their immutability-per-version pattern is exactly what we propose.
+- **Skip.** Dynamic Parameters (runtime-eval), template sharing/RBAC (enterprise), publish workflow.
+
+## 7. Database access — sqlc + `dbauthz` + paired migrations
+
+- **Where.** Queries: `coderd/database/queries/*.sql`. Wrapper: `coderd/database/dbauthz/dbauthz.go`. Wiring: `coderd/coderd.go:374-380`. Migrations: `coderd/database/migrations/` (paired up/down).
+- **What.** `.sql` files → `make gen` runs sqlc → typed Go. All access through `Store` interface, wrapped by `dbauthz.New()` enforcing authorization per query. Unauthorized = "not authorized" (no reason leak). Migrations paired = clean rollback.
+- **Why for us.** Phase 6. sqlc eliminates a class of runtime SQL bugs. The wrapper pattern is the cleanest implementation of "no DB query without auth context" we've seen.
+- **Skip.** `regosql` per-query auth (complex). Start with handler-side authz, then wrap.
+
+## 8. Streaming endpoints — `coder/websocket` lib + custom JSON encoder
+
+- **Where.** `coderd/workspaceagents.go:481-510, 855-870, 1227-1240`.
+- **What.** `github.com/coder/websocket` (not gorilla — newer/lighter); accept + wrap with JSON encoder; auto compression + ping/pong negotiation.
+- **Why for us.** Phase 6. Direct fit for our CDP / ttyd / MCP-streaming proxying.
+- **Skip.** Their DERP mesh (Tailscale-specific), workspace-agent multiplexing.
+
+## 9. CLI ↔ server — REST over HTTP with bearer tokens
+
+- **Where.** `cli/root.go`, `cli/login.go`, SDK: `codersdk/`, server auth: `coderd/httpmw/apikey.go`.
+- **What.** CLI is a standalone Go binary calling the **same** REST API as the frontend. Bearer token in `Authorization` header. No custom CLI protocol. SDK in `codersdk/` is importable by third parties.
+- **Why for us.** Phase 6. Sane default for future admin CLI — no separate gRPC just for the CLI. Public SDK is a real adoption multiplier.
+- **Skip.** `config-ssh` helper, agent connection pooling (workspace-SSH-specific).
+
+## 10. Project layout — flat `cli/`, `coderd/`, `codersdk/`
+
+- **Where.** Repo root.
+- **What.**
+  - `cli/` — commands.
+  - `coderd/` — control plane logic.
+  - `codersdk/` — shared request/response types + Go client (importable).
+  - Nested under `coderd/` by domain: `rbac/`, `audit/`, `database/`, `httpmw/`, `provisionerjobs/` etc.
+- **Why for us.** Phase 6. Directly adoptable layout. Public SDK at `codersdk/` matches our intent to keep the MCP contract first-class.
+- **Skip.** `enterprise/` directory (defer the commercial split).
+
+## 11. Testing — `coderdtest.New(t, nil)` harness with real Postgres
+
+- **Where.** `coderd/coderdtest/coderdtest.go:1-150` + helpers.
+- **What.** Spawns real Postgres + coderd instance + bootstraps templates/users → returns test client. Table-driven, `t.Parallel()`. Test patterns: create user → workspace → wait for build → assert.
+- **Why for us.** Phase 6. Real-DB integration tests catch race/lock/constraint bugs that mocks miss. Matches our existing integration-test posture (commit `7a55968`).
+- **Skip.** OIDC mocking, load harness, enterprise-only utilities.
+
+## 12. Secrets — encrypted in DB with key rotation
+
+- **Where.** `coderd/usersecrets.go:1-100`, `coderd/cryptokeys/`, rotator start at `coderd/coderd.go:588`, schema `coderd/database/queries/crypto_keys.sql`.
+- **What.** User secrets (API keys, OAuth tokens, env vars) stored encrypted with rotation-aware key. Old keys readable for decryption; new encryptions use latest key. Never logged/exported.
+- **Why for us.** Phase 4 (secret broker) + Phase 6. Pattern is directly applicable — single-DB storage, in-process key management, rotation as a background task.
+- **Skip.** Azure Key Vault (substitute our chosen secret backend), enterprise export.
+
+## Adoption priority for Phase 6 MVP
+
+1. Chi router + layered middleware (`coderd/coderd.go:500-1050`)
+2. API-key middleware with context population (`coderd/httpmw/apikey.go`)
+3. sqlc + `dbauthz` wrapper (`coderd/database/dbauthz/`)
+4. Simple RBAC (`coderd/rbac/authz.go`) — Rego later
+5. DB-backed audit log (`coderd/audit.go`)
+6. Provisioner-job abstraction for sandbox lifecycle
+7. `coderdtest`-style real-DB integration harness
+8. WebSocket streaming via `coder/websocket` (`coderd/workspaceagents.go`)
+
+**Defer to Phase 6.5+.** Multi-IDP sync, OAuth-provider mode, full Rego policies, template-version Dynamic Parameters, multi-key rotation.

package/docs/future-architecture/research/04-cloud-hypervisor.md

@@ -0,0 +1,99 @@
+<!-- SPDX-License-Identifier: FSL-1.1-Apache-2.0 -->
+<!-- Copyright (c) 2025 Open Computer Use Contributors -->
+
+# 04 — Cloud Hypervisor (lead untrusted-tier microVM)
+
+> Source: [cloud-hypervisor/cloud-hypervisor](https://github.com/cloud-hypervisor/cloud-hypervisor). Rust microVM hypervisor, Intel-led.
+> Primary backend for [Phase 9](../roadmap.md) (`kata-ch`); snapshot/restore feeds Phase 10.
+
+## 1. REST API on Unix socket — VM lifecycle
+
+- **Where.** `docs/api.md:51-59, 61-102` (OpenAPI 3.0). `--api-socket path=/tmp/ch.sock`.
+- **What.** Endpoints: `/vmm.ping`, `/vmm.shutdown`, `/vm.create`, `/vm.boot`, `/vm.pause`, `/vm.resume`, `/vm.snapshot`, `/vm.restore`, `/vm.shutdown`. Plus `/vm.resize` (CPU/memory hotplug) and `/vm.add-*` (disk/fs/net/vsock hotplug).
+- **Why for us.** Phase 9 control plane → hypervisor. No HTTP/2 / no streaming — keep it synchronous.
+- **Skip.** No authn in CH — rely on socket filesystem permissions; orchestrator wraps the socket.
+
+## 2. vsock — CID model + bidirectional setup
+
+- **Where.** `docs/vsock.md:1-19, 51-75`. Kernel: `CONFIG_VHOST_VSOCK` (host), `CONFIG_VIRTIO_VSOCKETS` (guest).
+- **What.** CIDs: Hypervisor=0, Loopback=1, Host=2, Guest=3+. Stream only. `--vsock cid=3,socket=/tmp/ch.vsock`.
+  - Host→Guest: guest listens on port, host connects via Unix socket with `CONNECT <port>` prefix (socat ≥1.7.4).
+  - Guest→Host: host listens on `<socket_path>_<port>`, guest connects to CID=2.
+- **Why for us.** Phase 7 vsock listener (`05-layer1-guest-agent.md`). The `CONNECT <port>` protocol detail is non-obvious; document in our agent spec.
+- **Skip.** Loopback CID=1 only needed for debug.
+
+## 3. virtio-fs — fast shared mounts (the reason we pick CH)
+
+- **Where.** `docs/fs.md:13-90`. Daemon: `virtiofsd` (separate Rust binary). VM needs `--memory shared=on` (mandatory).
+- **What.**
+  - Build virtiofsd separately; `setcap cap_sys_admin+epi`.
+  - `--fs tag=myfs,socket=/tmp/virtiofs,num_queues=1,queue_size=512`.
+  - Guest: `mount -t virtiofs myfs /mnt/shared` (kernel ≥5.10).
+  - Cache modes: `cache=never` (default; low RAM, dense) vs `cache=always` (faster, RAM-multiplier — **footgun** at high density).
+- **Why for us.** Phase 9 — replaces FUSE for skill / user-data mounts inside the microVM. Updates [`architecture/06-storage.md`](../architecture/06-storage.md) "What changes per phase" row for the kata tier (virtio-fs over FUSE).
+- **Skip.** DAX feature not stable; avoid.
+
+## 4. Snapshot / restore — files + ondemand restore
+
+- **Where.** `docs/snapshot_restore.md:11-144`.
+- **What.**
+  - Snapshot: pause VM → `POST /vm.snapshot {source_url: file:///path}` → produces `config.json`, `memory-ranges`, `state.json`.
+  - Restore: `cloud-hypervisor --restore source_url=file:///path,resume=true` OR `POST /vm.restore`. Restored VM is **paused** — must explicitly `/vm.resume`.
+  - `memory_restore_mode=ondemand` — userfaultfd-based; skips full-memory copy (faster boot). Fails strict if userfaultfd unavailable.
+- **Why for us.** Phase 10. Note: snapshot file size ≈ VM RAM size → 100 VMs × 1 GB = 100 GB fast storage.
+- **Skip.** VFIO devices **break** snapshot/restore. If we ever offer GPU passthrough, those templates have no snapshot capability.
+
+## 5. Memory — balloon, free-page reporting, ACPI hotplug
+
+- **Where.** `docs/balloon.md:8-76`, `docs/memory.md:64-86`, `docs/hotplug.md:64-75`.
+- **What.**
+  - **Balloon** (`--balloon size=...,deflate_on_oom=on,free_page_reporting=on`) — host reclaims guest pages.
+  - **Free Page Reporting** alone (even with balloon size 0) cuts host footprint without shrinking guest visible RAM. Best for high-density untrusted tier.
+  - **ACPI hotplug**: `/vm.resize` to grow; **shrink takes effect only on guest reboot** (footgun).
+  - Reserve headroom: `--memory size=1G,hotplug_size=2G`.
+- **Why for us.** Phase 5+ capacity policy. Free Page Reporting is the easy density win.
+- **Skip.** Hugepages — only if a workload is latency-critical; otherwise hurts packing.
+
+## 6. GPU passthrough — VFIO (future)
+
+- **Where.** `docs/vfio.md:1-150`.
+- **What.** Unbind from native driver → bind to `vfio-pci` → pass `--device path=/sys/bus/pci/devices/...`. NVIDIA P2P: `x_nv_gpudirect_clique=0`.
+  - IOMMU group: all devices in same group must be passed (or none).
+  - Snapshot incompatible with VFIO.
+- **Why for us.** Out of scope until Phase 10+, but documented now so we don't promise it for templates that need snapshotting.
+
+## 7. Privilege model — capability-based, no jailer
+
+- **Where.** `docs/seccomp.md:1-68`, `docs/landlock.md:1-106`.
+- **What.** **No** Firecracker-style jailer. Single process; only `cap_net_admin+ep` for TAP networking. Hardening = seccomp (per-thread allowlists, on by default, kill-on-violation) + **Landlock** sandboxing (Linux ≥5.13) for FS access. Hotplug paths must be pre-declared in `--landlock-rules`.
+- **Why for us.** Phase 9 — orchestrator wraps CH in a container/cgroup boundary; CH itself relies on seccomp + Landlock. Compare with [Firecracker's jailer](./05-firecracker.md) — different security model.
+- **Footguns.** Never `--seccomp false` in production; never run as root; Landlock + hotplug requires upfront path declaration.
+
+## 8. TDX / SEV-SNP attestation (compliance tier)
+
+- **Where.** `docs/intel_tdx.md:26-97`, `docs/amd_sev_snp.md:16-40`.
+- **What.** Build with `--features tdx` (Intel) or `--features sev_snp` (AMD, MSHV-only). Encrypted guest, hypervisor-blind. No balloon, no VFIO under TDX. 10–30 % perf penalty.
+- **Why for us.** Phase 10+ if a compliance tier (HIPAA/Confidential Computing) is added. ADR-worthy when it lands.
+
+## 9. Footguns — explicit
+
+| What NOT to do | Why | Fix |
+|---|---|---|
+| Snapshot + VFIO | Snapshot fails on VFIO devices | No GPU passthrough on snapshottable templates |
+| `cache=always` at density | Host page cache multiplies | Always `cache=never` for untrusted tier |
+| Landlock hotplug without pre-declaration | Add-disk denied | Pre-declare all possible hotplug paths |
+| `deflate_on_oom=on` w/o testing guest | Older Linux/Windows may not handle | Default off; test per-image |
+| Trust TDX report w/o attestation chain | Rogue hypervisor fakes it | Validate against Intel/AMD roots |
+| Run as root | Larger attack surface | `setcap cap_net_admin+ep` + seccomp |
+| `--seccomp false` in prod | Removes syscall allowlist | Keep on; use `--seccomp log` for debug |
+
+## Summary for Phase 9 (`kata-ch`)
+
+1. REST on unix socket — orchestrator control.
+2. vsock CID=3 for agent comms.
+3. virtio-fs with `cache=never` for skill / user-data mounts.
+4. Landlock + seccomp enabled by default.
+5. Headroom: `--memory size=1G,hotplug_size=2G` per template.
+6. Free Page Reporting on by default for density.
+7. **No** VFIO on snapshottable templates.
+8. Snapshot files sized = guest RAM — provision fast block storage.

package/docs/future-architecture/research/05-firecracker.md

@@ -0,0 +1,114 @@
+<!-- SPDX-License-Identifier: FSL-1.1-Apache-2.0 -->
+<!-- Copyright (c) 2025 Open Computer Use Contributors -->
+
+# 05 — Firecracker (kata-fc backend, fastest-cold-start tier)
+
+> Source: [firecracker-microvm/firecracker](https://github.com/firecracker-microvm/firecracker). AWS Lambda/Fargate microVM.
+> Backend for [Phase 9](../roadmap.md) `kata-fc` (free-trial / fastest-cold-start tier). Cloud Hypervisor remains primary; FC fills the "smallest attack surface" niche.
+
+## 1. REST API on unix socket — synchronous-only
+
+- **Where.** `src/vmm/src/rpc_interface.rs:50-150` (`VmmAction` enum), `src/firecracker/swagger/firecracker.yaml`.
+- **What.** HTTP/1.1 only (no HTTP/2). All actions are `VmmAction` variants. States: `NotStarted` → `Running` → `Paused` → snapshot/resume. Pause is mandatory before snapshot.
+- **Why for us.** Phase 9 alternative backend. No streaming → cleaner than CH for "give me a sandbox now" path.
+- **Skip.** No WebSocket / no async jobs.
+
+## 2. Jailer — privilege-drop wrapper
+
+- **Where.** `src/jailer/src/main.rs:1-250`, `src/jailer/src/chroot.rs:19-100`, `src/jailer/src/cgroup.rs`, `src/jailer/src/env.rs`.
+- **What.** Stateless wrapper binary that, before exec'ing firecracker, does:
+  - **Mount namespace isolation:** `unshare(CLONE_NEWNS)` → `pivot_root()` → umount old_root.
+  - **Chroot via bind-mount + pivot_root** (not naive `chroot()`).
+  - **uid/gid drop** (`--uid`, `--gid`).
+  - **Cgroup placement** (v1 & v2 — moves FC pid into a child cgroup).
+  - **Optional `--netns`** (network-ns inheritance).
+  - **Seccomp filters** loaded by firecracker itself after jailer execs it.
+- **Why for us.** This is the standard hypervisor-hardening pattern. Our Phase 9 Helm/CRD work for `kata-fc` templates should use jailer-equivalent containment, even if the orchestrator wraps CH instead.
+- **Skip.** Assumes host enforces file perms; no AppArmor/SELinux integration. `resource_limits` moved to cgroups in the 2024 release.
+
+## 3. MMDS — guest metadata service (EC2-compatible)
+
+- **Where.** `src/vmm/src/mmds/data_store.rs:1-230` (JSON store, 51.2 KB default limit), `src/vmm/src/mmds/ns.rs` (TCP), `src/vmm/src/mmds/token.rs` (V2 session tokens), `docs/device-api.md:26`.
+- **What.**
+  - In-process metadata server at `169.254.169.254` (EC2-compatible).
+  - **V1**: no auth, deprecated.
+  - **V2**: session token (HMAC), required for cross-tenant safety on shared hosts.
+  - JSON tree set via `PUT /mmds`, patched via `PATCH`.
+  - **Requires** a virtio-net device to enable.
+  - **Not persisted across snapshots** — config saved, data store cleared. Reconfigure on restore.
+- **Why for us.** Phase 9 — cheap, no-network bootstrap of guest config (env vars, JWT for L4 callback). Replaces a chunk of what we'd otherwise do through `/v1/configure` for VM-class templates.
+- **Skip.** No nested auth; V2 tokens are simple HMAC.
+
+## 4. Snapshot / restore — files + memory mmap + versioning
+
+- **Where.** `src/vmm/src/persist.rs:1-100` (`MicrovmState`), `docs/snapshotting/snapshot-support.md:32-172`, `src/vmm/src/rpc_interface.rs:67, 98`.
+- **What.**
+  - Files: (1) memory file (guest RAM), (2) vmstate (JSON + bincode + 64-bit CRC32), (3) disk files (user-managed).
+  - Restored via `MAP_PRIVATE` mmap → on-demand paging + COW.
+  - Versioning `MAJOR.MINOR.PATCH` — incompat versions rejected.
+  - Boot from snapshot < 125 ms.
+  - **vsock connections closed on snapshot** (listening sockets survive).
+  - **Network connection state not guaranteed.**
+  - No built-in encryption — user-managed at the storage layer.
+- **Why for us.** Phase 10. Snapshot file lifetime ≥ resumed VM lifetime.
+- **Caveat.** GIC version (aarch64) must match between snapshot and restore hosts.
+
+## 5. Memory & resource limits
+
+- **Where.** `src/vmm/src/vstate/memory.rs`, `src/vmm/src/resources.rs:1-100`, jailer cgroup integration.
+- **What.**
+  - Guest RAM as anonymous mmap (or hugepages if configured). 1 MiB → 32 TiB theoretical.
+  - **Oversubscription enabled by default** — host OOM killer can evict.
+  - 1–32 vCPUs per microVM.
+  - virtio-mem hotplug **advertised but not optimized** — don't rely on dynamic memory in practice.
+- **Why for us.** Phase 9 — predictable per-VM cost (~5 MiB VMM overhead). Densely pack idle microVMs.
+
+## 6. Seccomp filters
+
+- **Where.** `src/vmm/src/seccomp.rs:1-137`, `docs/seccomp.md:1-87`, `resources/seccomp/` (JSON rules).
+- **What.**
+  - **Compiled at build time** (`seccompiler-bin` → bitcode embedded in binary).
+  - Per-thread filters: vmm, api, vcpu (separate allowlists).
+  - Max 4096 BPF instructions/filter.
+  - Override at runtime: `--seccomp-filter <path>`.
+  - **Never disable in prod** (`--no-seccomp`).
+- **Why for us.** Phase 9 — direct lesson: per-thread allowlists. Don't write one big filter; segment by thread role.
+
+## 7. Logging & metrics — named-pipe drain, best-effort
+
+- **Where.** `src/vmm/src/logger/`, `docs/metrics.md`, `src/vmm/src/rpc_interface.rs:60-62`.
+- **What.** Plain-text logs to named pipe. Metrics every 60 s + on events (start, panic). Counters for per-device I/O, vCPU halts, `lost-logs`, `lost-metrics` (when pipe full).
+- **Why for us.** Phase 9 — must drain the pipe in real time or signals are lost. Our metrics shipper must keep up.
+- **Skip.** No structured logging (no syslog/JSON). Customer owns aggregation.
+
+## 8. What Firecracker explicitly does NOT support
+
+This list is the reason **Cloud Hypervisor is our primary** and FC is the secondary backend:
+
+- **✗ virtio-fs** — block devices + vsock only. → Use CH for skill / user-data mounts.
+- **✗ GPU / VFIO** — no IOMMU, no GPU paravirt. → CH or kata-qemu for GPU.
+- **✗ Arbitrary PCI hotplug** — only virtio-block/net/pmem hotplug as developer-preview.
+- **✗ Guest graceful reboot (x86_64)** — only ARM64 via PSCI.
+- **✗ Nested virt.**
+- **✗ 32-bit guest.**
+- **✗ ACPI PM / thermal throttling (x86).**
+- **Devices total**: virtio-net, virtio-block, virtio-balloon, virtio-vsock, serial, minimal i8042.
+
+## 9. Multi-arch (x86_64 ↔ aarch64)
+
+- **Where.** `src/arch/x86_64/`, `src/arch/aarch64/`.
+- **What.** Both first-class. Tested on AWS Intel/AMD/Graviton metals. GICv2/GICv3 supported on aarch64 but **snapshot/restore requires same GIC version** on both hosts.
+
+## Summary table
+
+| Pattern | File | Phase | Constraint |
+|---|---|---|---|
+| REST on unix socket | `rpc_interface.rs` | 8 | Synchronous only |
+| **Jailer** (chroot + ns + cgroup + uid) | `jailer/src/main.rs` + `chroot.rs` | 8 | Standard hardening pattern — adopt principle |
+| MMDS V2 (token-auth) | `mmds/data_store.rs` | 8 | Cleared on snapshot restore |
+| Snapshot files + CRC + ondemand | `persist.rs` | 10 | Memory file must persist |
+| Memory oversubscription | `vstate/memory.rs` | 8 | OOM killer can evict |
+| Per-thread seccomp BPF | `seccomp.rs` | 8 | Compiled in; segmented by role |
+| Named-pipe logs/metrics | `logger/` | 8,10 | Must drain in real-time |
+| Constraint: no virtio-fs/GPU | `docs/design.md` | 8 | Use CH for those workloads |
+| Multi-arch | `arch/` | 8,10 | Snapshot needs matching GIC on aarch64 |