@dmgnr/kuber 1.3.0 → 2.1.0

package/README.md

@@ -1,45 +1,280 @@
 # kuber
 
-`kuber` is Astral's internal Docker Compose to Kubernetes translation layer.
-
-It reads a local Compose file, renders Kubernetes resources, applies them to the cluster, and can build images remotely from your working tree without requiring Docker on your machine.
-
-## What It Does
-
-- Reads `compose.yml` or `docker-compose.yml`
-- Converts supported Compose services into Kubernetes resources
-- Applies those resources into a namespace named after the current directory
-- Turns `env_file` entries into Kubernetes Secrets and mounts them through `envFrom`
-- Translates file mounts into ConfigMaps and directory/volume mounts into PVC-backed volumes
-- Builds images on the remote builder by syncing:
-  - committed git state
-  - tracked local diffs
-  - untracked files
-  - ignored `.env*` files
-- Supports host-based `ports` syntax that renders Kubernetes `Ingress` rules
-- Supports managed Postgres claims through special pseudo-volumes such as `postgresql:app`
-- Supports managed S3 buckets and credentials through pseudo-volumes such as `s3:app`
+`kuber` is a Docker Compose to Kubernetes translation layer backed by a
+self-hosted management service (`kuber-server`). It reads a local Compose file,
+renders Kubernetes resources, and applies them to the cluster through an
+authenticated v2 API. Image builds happen inside the cluster with rootless
+BuildKit, so the workstation needs neither Docker nor `kubectl`.
 
-## Environment Assumptions
+## Architecture
+
+```
+workstation (kuber CLI)  ──HTTPS──▶  kuber-server (in-cluster pod)
+                                        │
+                                        ├─ CAS (content-addressed blobs on RWX PVC)
+                                        ├─ BuildKit Jobs (rootless) ──▶ registry
+                                        └─ Kubernetes API (RBAC-scoped)
+```
 
-This tool is built for Astral's cluster and workstation setup. It is not intended to work unchanged outside that environment.
+The CLI authenticates to the server with a Bearer token acquired by `kuber
+login`. It snapshots the repository into a content-addressed workspace, uploads
+only the blobs the server is missing, and submits a build request. The server
+materializes that workspace onto a shared RWX PVC and runs a rootless BuildKit
+Job that builds and pushes the image, then the CLI pins the resulting immutable
+registry digest into the rendered Deployment.
 
-Expected local setup:
+The server also owns reconciliation: it plans, applies, and prunes resources in
+a per-project namespace, reconciles managed Postgres and S3 claims, rolls
+deployments back, streams logs, and exposes interactive `exec` sessions over a
+WebSocket.
 
-- Tailscale access to the remote builder and Kubernetes network
-- a working `~/.kube/config`
-- `ssh` available locally
+## Environment Assumptions
+
+kuber targets a specific self-hosted cluster and workstation setup. It is not
+intended to run unchanged against an arbitrary Kubernetes environment. The
+expected setup:
+
+- an account on the `kuber-server` management API
+- a working ClusterRole/Role (see [Server Deployment](#server-deployment))
+- a registry the in-cluster BuildKit can push to
 
 Not required locally:
 
 - Docker
 - `kubectl`
 
-Docker is not required because builds happen on the remote builder after `kuber` syncs your repo state there. `kubectl` is not required because cluster access is handled through the bundled Kubernetes client.
+## API Origin
+
+The v2 management API has one hard-coded origin:
+
+```text
+https://kuber.astrxl.dev/api/v2
+```
+
+## Authentication
+
+```bash
+kuber login dmgnr
+kuber login dmgnr --persist
+kuber whoami
+kuber logout
+```
+
+The default login is stored with mode `0600` under
+`$XDG_RUNTIME_DIR/kuber/session.json` and disappears with the user runtime
+directory. `--persist` instead uses `$XDG_CONFIG_HOME/kuber/session.json`, or
+`~/.config/kuber/session.json` when `XDG_CONFIG_HOME` is unset. Password input
+is never echoed. Runtime sessions last 24 hours and persistent sessions last 30
+days; `logout` revokes the server-side session. `readSession` falls back to the
+persistent file when no runtime session exists.
+
+The default login is scoped to the current user and the authenticated identity
+is available to every v2 command. `login`, `logout`, and `whoami` are the only
+commands that run without a loaded project configuration.
+
+### Roles and Authorization
+
+The server grants capabilities through three roles:
+
+- `viewer` — read-only cluster access (`kubernetes:read`)
+- `operator` — `viewer` plus `kubernetes:write` and `kubernetes:exec`
+- `admin` — all capabilities, including user administration
+  (`users:read`, `users:write`, `sessions:revoke`)
+
+Administer users with the `users` command tree:
+
+```bash
+kuber users ls
+kuber users add dmgnr --roles admin
+kuber users update dmgnr --roles operator
+kuber users update dmgnr --password
+kuber users disable dmgnr
+kuber users enable dmgnr
+kuber users delete dmgnr
+kuber users revoke dmgnr
+```
+
+`users add`, `update --password`, and `delete` prompt for password / written
+confirmation on an interactive terminal. Passwords are hashed with Argon2id on
+the server and never stored in plaintext. Updating a user's roles or password
+revokes all of that user's active sessions.
+
+Inspect server-side operations and the audit trail:
+
+```bash
+kuber operations ls
+kuber operations get <operation-id>
+kuber audit ls
+```
+
+## Server Deployment
+
+The repository's `compose.yml` owns the `kuber-system` namespace, the
+`kuber-server` image, its Deployment, Service, and Ingress. `.kuberrc.ts`
+extends the rendered manifests with the server's ServiceAccount and RBAC:
+
+- a namespaced `Role`/`RoleBinding` (`kuber-server-auth`) for the `kuber-system`
+  Secrets, ConfigMaps, Pods, Jobs, and Leases the server itself reads and writes
+- a `ClusterRole`/`ClusterRoleBinding` (`kuber-server-manager`) granting the
+  cross-namespace verbs it needs to manage user projects
+
+`compose.yml` runs the server as a non-root user (`runAsUser`/`runAsGroup`
+`1000`), drops all Linux capabilities, uses a read-only root filesystem with a
+`RuntimeDefault` seccomp profile, and backs `/data` with a Longhorn PVC
+(`kuber-build-data`). The PVC hosts the CAS, materialized workspaces, and
+resumable upload bytes, and is shared with BuildKit Jobs.
+
+Deploy the server with kuber itself:
+
+```bash
+KUBER_BOOTSTRAP_PASSWORD='replace-me' kuber up
+```
+
+The server creates an `admin` user (from `KUBER_BOOTSTRAP_USERNAME`, default
+`dmgnr`) on first boot only if that user does not already exist. The bootstrap
+Secret is only rendered while `KUBER_BOOTSTRAP_PASSWORD` is set. After logging
+in successfully, reconcile without that variable and restart once so kuber
+removes the stale bootstrap Secret and the password leaves the pod environment:
+
+```bash
+kuber up --no-b
+kuber restart kuber-server
+```
+
+See [Account Recovery](#account-recovery) for what to do if you are locked out.
+
+### Security Constraints
+
+Because the server's ServiceAccount is scoped by the RBAC in `.kuberrc.ts`, it
+can only act on the resources kuber manages. The management layer additionally
+enforces ownership, so the server refuses to mutate resources that do not carry
+kuber's workspace labels. Deleting a resource requires its UID as an optimistic
+concurrency precondition, and workspace deletion also requires the namespace
+UID. Namespaces that are not owned by kuber, or owned by a different workspace,
+are never mutated (see [Workspaces and Migration](#workspaces-and-migration)).
+
+## Workspaces and Migration
+
+Each project maps to a Kubernetes namespace derived from the current working
+directory. The CLI records a "workspace" on the server keyed by project name and
+borrows the namespace UID to guarantee it owns the namespace before reconciling.
+Adopting existing resources relabels them (Server-Side Apply) only when they are
+already managed by kuber and not owned by another workspace.
+
+`up` refuses to mutate a namespace when:
+
+- the namespace already exists but does not carry kuber's managed-by label
+  (`external`), or
+- the namespace is labeled for a different workspace UID
+  (`different-workspace`)
+
+In those cases the CLI prints a hint to `POST
+/workspaces/<project>/adopt` with the namespace UID to complete a safe,
+explicit adoption. The platform namespace (`kuber-system`) is adopted through
+the admin-only platform adoption route.
+
+Workspace state, revisions, operations, and audit events are stored as Secrets
+and ConfigMaps in `kuber-system` keyed by kuber's `kuber.astrxl.dev/type`
+label. Workspace updates are optimistic (If-Match on resource version) and
+immutable revisions are recorded so history survives. Expired sessions are
+cleaned up on an interval, and stale operations are marked failed on server
+startup recovery.
+
+### Migration Behavior
+
+`kuber up` is idempotent and safe to re-run. Each run:
+
+1. snapshots the workspace and uploads missing blobs to the server CAS,
+2. ensures the workspace record (creating or updating it with an optimistic
+   If-Match),
+3. adopts the namespace and its kuber-managed resources,
+4. reconciles managed Postgres and S3 claims,
+5. renders manifests, plans the diff, applies desired resources, waits for
+   rollout, and deletes stale resources.
+
+Because resource references are immutable digests, re-running `up` only restarts
+deployments whose image content actually changed. `start` re-resolves published
+digests without building.
+
+## Registry Authentication
+
+Building and pushing images from inside the cluster typically requires
+credentials for the target registry. These are read from a Docker config file
+(`KUBER_REGISTRY_CONFIG`, default `/etc/kuber/registry/config.json`) and mounted
+as an image pull secret named by `KUBER_REGISTRY_SECRET`.
+
+Registry authentication is optional. If `KUBER_REGISTRY_SECRET` is unset, the
+server warns at startup and BuildKit uses anonymous registry access. This is
+intended for registries that allow anonymous push/pull. The same credentials are
+used when the server resolves a published image digest (`start` / `export`).
+
+The server supports both standard registry bearer-token (`WWW-Authenticate:
+Bearer`) and pre-emptive Basic auth when resolving digests.
+
+### Build Registry Environment
+
+The server's registry behavior is driven by a few closely related environment
+variables:
+
+- `KUBER_BUILD_REGISTRY` (default `registry.neko-piranha.ts.net`): the registry
+  the in-cluster BuildKit pushes built images to and `kuber` uses as the image
+  namespace. It also supplies the registry host for authentication.
+- `KUBER_INTERNAL_REGISTRY_HOST`: the host of an internal registry (for example
+  the in-cluster distribution service) used for the BuildKit cache image and,
+  when set, the image direct push target. When unset, push and cache fall back
+  to `KUBER_BUILD_REGISTRY`.
+- `KUBER_INTERNAL_REGISTRY_INSECURE`: set to `"true"` to push to the internal
+  registry over plain HTTP instead of HTTPS. Only meaningful when
+  `KUBER_INTERNAL_REGISTRY_HOST` is set.
+- `KUBER_PUSH_IMAGE_PREFIX` (default `kuber/`): a prefix applied to project
+  images pushed to the internal registry when `KUBER_INTERNAL_REGISTRY_HOST` is
+  configured.
+- `KUBER_REGISTRY_RESOLVE_ORIGIN`: an explicit origin used to resolve a
+  published image digest (used by `start` / `export`). Useful when the digest
+  must be resolved from a different endpoint than the build/push registry, such
+  as an internal HTTP registry.
+- `KUBER_REGISTRY_CONFIG`: path to the Docker config file with registry
+  credentials (see above).
+- `KUBER_REGISTRY_SECRET`: the image pull Secret mounted for BuildKit's
+  registry access.
+- `KUBER_BUILDKIT_IMAGE`: override the BuildKit runner image used for builds.
+- `KUBER_BUILD_DATA_CLAIM`: the PVC claim backing builds.
+
+See [Server Deployment](#server-deployment) for how `compose.yml` wires the
+internal registry variables for the kuber-server pod.
+
+## Account Recovery
+
+If you lose your credentials and cannot log in:
+
+1. Recreate the bootstrap admin by deploying with
+   `KUBER_BOOTSTRAP_PASSWORD` set again:
+   ```bash
+   KUBER_BOOTSTRAP_PASSWORD='new-password' kuber up --no-b
+   kuber restart kuber-server
+   ```
+2. The server only creates the bootstrap user if the account does not already
+   exist, so a fresh `kuber login <username>` with the new password works, or
+   use the newly created admin to reset other accounts:
+   ```bash
+   kuber users update <username> --password
+   ```
+3. After recovering, reconcile without `KUBER_BOOTSTRAP_PASSWORD` and restart
+   so the bootstrap Secret is removed and the password leaves the pod
+   environment.
+
+Because user records and session hashes are stored as Secrets in
+`kuber-system`, recovery relies on cluster administrators being able to redeploy
+the server with bootstrap credentials. `users revoke <username>` forcibly logs a
+user out across all devices.
 
 ## Next.js Example
 
-[`example/`](example/) contains a documented deployment template for adding kuber to an existing Bun-powered Next.js project without initializing or bundling an application in this repository. It includes a standalone-output Dockerfile, `.dockerignore`, `compose.yml`, and the required Next.js configuration.
+[`example/`](example/) contains a documented deployment template for adding
+kuber to an existing Bun-powered Next.js project without initializing or
+bundling an application in this repository. It includes a standalone-output
+Dockerfile, `.dockerignore`, `compose.yml`, and the required Next.js
+configuration.
 
 ## Running
 
@@ -72,16 +307,23 @@ bun run index.ts db ls
 bun run index.ts s3 ls
 bun run index.ts s3 creds app
 bun run index.ts s3 ui app
+bun run index.ts login dmgnr
+bun run index.ts users ls
+bun run index.ts operations ls
+bun run index.ts audit ls
 ```
 
 All commands accept `--config` to use a configuration file other than
 `.kuberrc.ts`:
 
 ```bash
-kuber --config deploy/production.kuberrc.ts up
-kuber up --config deploy/production.kuberrc.ts
+kuber --config production.kuberrc.ts up
+kuber up --config production.kuberrc.ts
 ```
 
+`login`, `logout`, and `whoami` are context-free and do not require a
+configuration.
+
 ### Shell Completion
 
 Generate and load completions for your shell:
@@ -91,27 +333,44 @@ source <(kuber complete zsh)
 source <(kuber complete bash)
 ```
 
-For a permanent setup, write the generated script to a file and source it from your shell configuration. Fish and PowerShell are also supported through `kuber complete fish` and `kuber complete powershell`.
+For a permanent setup, write the generated script to a file and source it from
+your shell configuration. Fish and PowerShell are also supported through
+`kuber complete fish` and `kuber complete powershell`.
 
 ## Commands
 
-- `up`: build images if needed, pin matching registry digests, render manifests, apply them, and wait for rollout
-- `start`: same as `up` but resolves the currently published image digests instead of building
-- `stop`: scale managed deployments to zero
-- `restart`: roll out a restart across managed deployments
-- `rollback` (alias `fuck`): roll one deployment back to its previous release, or all managed deployments when no name is given
-- `down`: delete managed resources while keeping ingress, PVCs, managed databases, and managed S3 storage
-- `down -f`: also delete ingress, PVCs, managed database and S3 resources, and the namespace
-- `ps`: print an ANSI tree of the current project namespace, hiding stopped deployments by default
-- `ps -a`: include stopped deployments and stale ReplicaSets in the tree
-- `logs [deployment]`: print logs for one deployment or all managed deployments
-- `logs -f [deployment]`: follow logs continuously
-- `exec <deployment> <command...>`: execute a command inside a running deployment pod
-- `db ls`: list managed Postgres claims declared in the current Compose file
-- `db creds <service>`: print the generated connection details for a managed Postgres claim
-- `s3 ls`: list managed S3 claims declared in the current Compose file
-- `s3 creds <service>`: print all generated S3 environment variables for a service
-- `s3 ui <service>`: print the Garage UI object-browser URL for a service bucket
+- `up [--no-b]`: build images if needed, ensure the workspace, reconcile managed
+  Postgres/S3, render manifests, apply them, wait for rollout, and delete stale
+  resources
+- `start`: like `up` but re-resolves the currently published image digests
+  instead of building
+- `login [username] [--persist]`: authenticate with the kuber API
+- `logout`: revoke and remove the current API session
+- `whoami`: show the authenticated API user and roles
+- `users`: administer user accounts and roles
+- `operations`: inspect server-side reconciliation operations
+- `audit`: inspect the audit trail
+- `ps [-a]`: print an ANSI graph of the current project namespace, hiding stopped
+  deployments by default
+- `logs [deployment] [-f]`: print (or follow) logs for one deployment or all
+  managed deployments
+- `exec <deployment> <command...>`: execute a command inside a running
+  deployment pod over an interactive WebSocket
+- `restart [deployment]`: roll out a restart across managed deployments
+- `stop`: delete the matching name-scoped HPAs (so autoscaling cannot scale
+  replicas back up) and scale managed deployments to zero
+- `rollback` (alias `fuck`) `[deployment]`: roll one deployment back to its
+  previous release, or all managed deployments when no name is given
+- `down [-f]`: delete managed resources while keeping ingress, PVCs, managed
+  databases, and managed S3 storage; `-f` also deletes those and the namespace
+- `db ls` / `db creds <service>`: list or print credentials for managed Postgres
+  claims
+- `s3 ls` / `s3 creds <service>` / `s3 ui <service>`: list managed S3 claims,
+  print their credentials, or print the Garage UI URL for a bucket
+- `export [-o file]`: render manifests to a YAML file without applying them
+
+Lifecycle commands (`restart`, `stop`, `rollback`, `down`) are idempotent:
+identical requests are deduplicated server-side and tracked as operations.
 
 ## Configuration
 
@@ -125,11 +384,6 @@ export default {
   project: "my-app",
   composeFile: "compose.production.yml",
   registry: "registry.example.com",
-  builders: {
-    amd64: "kuber@amd-builder",
-    arm64: "kuber@arm-builder",
-    remoteRoot: "kuber-build",
-  },
   rolloutTimeoutMs: 10 * 60_000,
 
   async compose(compose) {
@@ -147,37 +401,50 @@ export default {
 
 Operational defaults:
 
-- `project`: Compose top-level `name`, falling back to the current working directory name
+- `project`: Compose top-level `name`, falling back to the current working
+  directory name
 - `composeFile`: the first recognized Compose filename in the working directory
 - `registry`: `registry.neko-piranha.ts.net`
-- `builders.amd64`: `kuber@astral-th`
-- `builders.arm64`: `kuber@astral`
-- `builders.remoteRoot`: `kuber-build`
 - `rolloutTimeoutMs`: `300000`
 
 Project-name precedence is `.kuberrc.ts project`, Compose top-level `name`, then
 the current working directory name.
 
+The `registry` value controls which registry the CLI requests build images from
+and which the server uses to resolve published digests. `rolloutTimeoutMs`
+bounds how long `up` and `rollback` wait for a Deployment rollout.
+
 Configuration hooks can be synchronous or asynchronous and receive mutable
 values:
 
-- `compose(compose, context)`: once after parsing and validation; affects every command that reads Compose
-- `preBuild(compose, context)`: before build eligibility is evaluated when builds are enabled
-- `postBuild(result, context)`: after images are built; `result` contains `built` and `changed` service names
-- `postRender(resources, context)`: after rendering and before reconciliation planning; also runs for `export`
-- `postApply(resources, context)`: after desired resources are successfully applied
+- `compose(compose, context)`: once after parsing and validation; affects every
+  command that reads Compose
+- `preBuild(compose, context)`: before build eligibility is evaluated when
+  builds are enabled
+- `postBuild(result, context)`: after images are built; `result` contains
+  `built` and `changed` service names
+- `postRender(resources, context)`: after rendering and before reconciliation
+  planning; also runs for `export`
+- `postApply(resources, context)`: after desired resources are successfully
+  applied
 
 Hook context contains the resolved `cwd`, `project`, `composeFile`, and optional
 `configFile`. A hook error aborts the command and is reported by the normal CLI
 error handler.
 
+Registry and rollout configuration remain part of the CLI-facing contract.
+Image builder selection is not configurable: builds are scheduled, executed, and
+owned entirely by the server.
+
 ## Compose Conventions
 
-`kuber` supports a few project-specific Compose conventions on top of normal service translation.
+`kuber` supports a few project-specific Compose conventions on top of normal
+service translation.
 
 ### Host-Based Ports
 
-If a `ports` entry uses a hostname instead of a numeric published port, `kuber` treats it as an ingress host and routes traffic to the target container port.
+If a `ports` entry uses a hostname instead of a numeric published port, `kuber`
+treats it as an ingress host and routes traffic to the target container port.
 
 Example:
 
@@ -188,9 +455,11 @@ services:
       - somedomain.astrxl.dev:3000
 ```
 
-That produces a Kubernetes `Ingress` rule for `somedomain.astrxl.dev` pointing at the service port for container port `3000`.
+That produces a Kubernetes `Ingress` rule for `somedomain.astrxl.dev` pointing
+at the service port for container port `3000`.
 
-Single-level wildcard subdomains are supported. Quote wildcard entries so YAML does not treat the leading `*` as an alias:
+Single-level wildcard subdomains are supported. Quote wildcard entries so YAML
+does not treat the leading `*` as an alias:
 
 ```yml
 services:
@@ -200,7 +469,8 @@ services:
       - "*.secure.astrxl.dev:3001:protected"
 ```
 
-Protected routes use the kuber dialect and render Traefik `IngressRoute` resources instead of plain Kubernetes `Ingress`:
+Protected routes use the kuber dialect and render Traefik `IngressRoute`
+resources instead of plain Kubernetes `Ingress`:
 
 ```yml
 services:
@@ -216,6 +486,39 @@ Translation rules:
 - `host:port:protected` -> Traefik `IngressRoute` with middleware `routing/cf-auth` and host-wide matching
 - `host:port:protected(path1,path2,...)` -> Traefik `IngressRoute` with middleware `routing/cf-auth` and explicit `PathPrefix(...)` matches only
 
+### Replicas and Autoscaling
+
+`deploy.replicas` (or the top-level `scale` field) controls the Deployment
+replica count. A plain integer or numeric string renders a fixed `replicas`
+value, with `scale` taking precedence over `deploy.replicas`.
+
+A `"min-max"` range string requests autoscaling instead of a fixed count:
+
+```yml
+services:
+  app:
+    image: app
+    deploy:
+      replicas: "2-6"
+```
+
+kuber renders:
+
+- a `Deployment` with `replicas` set to the range minimum (`2`)
+- a `HorizontalPodAutoscaler` (`autoscaling/v2`) targeting that Deployment,
+  with `minReplicas: 2`, `maxReplicas: 6`, and a CPU target of 80%
+  utilization
+- a `100m` CPU request injected into the container, unless `x-container`
+  already specifies a CPU request (the HPA needs a CPU request to scale on)
+
+For both fixed counts above one and autoscaled ranges whose maximum exceeds
+one, kuber also adds a `topologySpreadConstraints` entry spreading pods across
+hosts (`kubernetes.io/hostname`, `maxSkew: 1`,
+`whenUnsatisfiable: ScheduleAnyway`).
+
+Malformed non-numeric replica values (for example `"lots"`) are rejected with
+a clear error instead of silently defaulting.
+
 ### Managed Postgres
 
 You can declare a managed Postgres database with a pseudo-volume:
@@ -236,7 +539,12 @@ services:
       - postgresql:user/database
 ```
 
-This creates or reuses the managed CNPG role secret, reconciles the database resource, and injects `DATABASE_URL` into the generated app secret in Kubernetes.
+This creates or reuses the managed CNPG role secret, reconciles the database
+resource, and injects `DATABASE_URL` and
+`REDIS_URL=redis://redis.database.svc.cluster.local` into the generated app
+secret in Kubernetes. `REDIS_URL` is only added to services with a managed
+Postgres claim. Running `kuber db creds <service>` performs the same focused
+Secret, role, and Database reconciliation before printing credentials.
 
 ### Managed S3
 
@@ -249,7 +557,8 @@ services:
       - s3:app
 ```
 
-This creates `GarageBucket/app` and `GarageKey/app` in `garage-system`. To use different key and bucket names:
+This creates `GarageBucket/app` and `GarageKey/app` in `garage-system`. To use
+different key and bucket names:
 
 ```yml
 services:
@@ -258,7 +567,8 @@ services:
       - s3:app-key/shared-assets
 ```
 
-The Garage operator generates the credentials. `kuber` reads its generated Secret and injects these values into the service's `<service>-env` Secret:
+The Garage operator generates the credentials. `kuber` reads its generated
+Secret and injects these values into the service's `<service>-env` Secret:
 
 - `AWS_ACCESS_KEY_ID`
 - `AWS_SECRET_ACCESS_KEY`
@@ -266,7 +576,9 @@ The Garage operator generates the credentials. `kuber` reads its generated Secre
 - `AWS_REGION`
 - `S3_BUCKET`
 
-One service can declare both `postgresql:...` and `s3:...`; all generated values are merged into the same service Secret. Managed Garage buckets and keys are retained by normal `down` and deleted by `down -f`.
+One service can declare both `postgresql:...` and `s3:...`; all generated
+values are merged into the same service Secret. Managed Garage buckets and keys
+are retained by normal `down` and deleted by `down -f`.
 
 Inspect a claim, print its generated credentials, or get its Garage UI URL:
 
@@ -282,9 +594,11 @@ assignments. `s3 ui` only prints the URL; it does not open a browser.
 
 ### Environment Files
 
-`env_file` entries are read locally and turned into a Kubernetes `Secret` named `<service>-env`. Deployments then consume that secret through `envFrom`.
+`env_file` entries are read locally and turned into a Kubernetes `Secret` named
+`<service>-env`. Deployments then consume that secret through `envFrom`.
 
-This is also where generated values such as `DATABASE_URL` and the managed S3 environment are injected.
+This is also where generated values such as `DATABASE_URL` and the managed S3
+environment are injected.
 
 ### Volumes
 
@@ -297,7 +611,11 @@ This is also where generated values such as `DATABASE_URL` and the managed S3 en
 - `postgresql:...` is treated as a managed database claim, not as a filesystem mount
 - `s3:...` is treated as a managed object-storage claim, not as a filesystem mount
 
-Named volumes also support kuber-specific storage hints.
+Named volumes also support kuber-specific Longhorn storage hints. Kuber renders
+each distinct placement policy as a deterministic, reusable Longhorn
+`StorageClass`, then references that class from the PVC. The generated class
+uses Longhorn's `numberOfReplicas`, `diskSelector`, and `dataLocality`
+parameters; placement fields are never written directly to the PVC.
 
 Default behavior:
 
@@ -329,8 +647,8 @@ services:
 Meaning:
 
 - `name(20Gi):/path` -> PVC size `20Gi`
-- `name(20Gi on archive):/path` -> PVC size `20Gi`, `diskTag: ["archive"]`, `dataLocality: "none"`
-- `name(20Gi on 1 archive):/path` -> PVC size `20Gi`, `replicaCount: 1`, `diskTag: ["archive"]`, `dataLocality: "none"`
+- `name(20Gi on archive):/path` -> PVC size `20Gi`, with a StorageClass using `diskSelector: "archive"` and disabled data locality
+- `name(20Gi on 1 archive):/path` -> PVC size `20Gi`, with a StorageClass using one replica and `diskSelector: "archive"`
 
 Explicit extensions are also supported.
 
@@ -368,14 +686,32 @@ Precedence:
 - top-level `volumes.<name>.x-*`
 - fallback default `1Gi on 2 fast`
 
+StorageClasses are cluster-scoped and content-addressed by policy. They are
+shared across projects and intentionally retained when a project is removed.
+
 ## Building
 
-Image builds run through the selected SSH builder. After a push, the registry's
-manifest digest is captured and embedded into the rendered Deployment as an
-immutable `:latest@sha256:...` reference, so a changed image naturally triggers
-a rollout (there is no separate "restart changed deployments" step). `start` and
-`export` look up the currently published digest without rebuilding, and fail if
-a buildable service has no published image yet.
+When a service declares `build`, the CLI:
+
+1. snapshots the repository into a content-addressed workspace
+   (committed git state, tracked changes, untracked files, and ignored `.env*`
+   files),
+2. negotiates with the server and uploads only the blobs it is missing,
+3. submits a build request; the server materializes the workspace from the CAS
+   onto a shared RWX PVC and runs a rootless BuildKit Job that builds and pushes
+   the configured image with registry cache.
+
+Build containers are restricted: they run as non-root (`runAsUser`/`runAsGroup`
+`1000`), do not mount the service account token, and only read the workspace
+(read-only mount) and a writable BuildKit state `emptyDir`. After a push, the
+registry's manifest digest is captured and embedded into the rendered Deployment
+as an immutable `:latest@sha256:...` reference, so a changed image naturally
+triggers a rollout. `start` and `export` look up the currently published digest
+without rebuilding, and fail if a buildable service has no published image yet.
+
+`export` is side-effect-free: it refuses to render managed Postgres or S3 claims
+(because it cannot call the server to generate credentials) and instead tells
+you to run `kuber up` or remove the managed provider claims.
 
 ## Rollback
 
@@ -395,23 +731,35 @@ Notes and limitations:
 - A later `kuber up` or `kuber start` re-resolves `:latest` and returns the
   Deployment to the current desired state anyway, so rollback is the right tool
   for responding to a bad deploy, not for permanently pinning an old version.
-- Rollback only considers Deployments managed by kuber (`app.kubernetes.io/managed-by=kuber`).
+- Rollback only considers Deployments managed by kuber
+  (`app.kubernetes.io/managed-by=kuber`) and only runs within a workspace whose
+  namespace kuber owns.
 
-To build distributable binaries:
+## Workspace State and Operations
 
-```bash
-bun run build.ts
-```
+The server persists per-workspace state, revisions, operations, and audit
+events as Secrets/ConfigMaps in `kuber-system`. Mutations are idempotent:
+
+- operations carry an idempotency key so retries do not double-apply,
+- resource deletion requires a UID precondition,
+- workspace replacement is guarded by If-Match.
 
-If you only want a plain JavaScript bundle for quick local use in another workspace, build `index.ts` directly:
+The CLI is built with the normal `build` script for distribution and local
+use:
 
 ```bash
-bun build index.ts --target bun --minify --sourcemap --outdir dist
+bun run build
 ```
 
 ## Notes
 
 - Resource names and namespaces are derived from the current working directory.
-- The remote build flow is optimized for local iteration, not for producing a perfectly clean export of the repository.
-- Managed database support is Kubernetes-only. It injects `DATABASE_URL` into the generated app secret and does not rewrite local `.env` files.
-- `kuber` operates on managed resources in the namespace matching the current directory name.
+- The workspace snapshot is optimized for local iteration, not for producing a
+  perfectly clean export of the repository.
+- Managed database support is Kubernetes-only. It injects `DATABASE_URL` into
+  the generated app secret and does not rewrite local `.env` files.
+- `kuber` operates on managed resources in the namespace matching the current
+  directory name.
+- Builds are scheduled, executed, and owned by the server. There is no local
+  SSH/daemon builder configuration; `registry` and `rolloutTimeoutMs` remain
+  CLI-facing settings.