kampodine 0.4.0 → 0.6.0

package/CHANGELOG.md

@@ -1,5 +1,123 @@
 # Changelog
 
+## 0.6.0 — 2026-10-08
+
+Three surfaces (per-instance profiles, VM metrics, OCI backups) plus
+host init auto-detection groundwork. Tests grew from 171 to 277.
+
+- **Profiles (per-instance config)** — `kampodine config
+  init|list|show|set-default|remove` manages `~/.kampodine/config.json`
+  (dir 0700, file 0600). FLAT by owner decision: each profile carries
+  `host` / `sshKey` (PATH only — the config never holds secrets) /
+  `proxyHost` / `group`; `group` is a cosmetic `config list --group`
+  filter, and the schema only LEAVES ROOM for a future `defaults` block —
+  no inheritance logic exists. The first profile auto-becomes the default;
+  removing the LAST profile requires `--force`; removing the default
+  promotes the alphabetically-first remaining profile. Resolution
+  EVERYWHERE (deploy, status, bluegreen, migrate, vm-prepare,
+  image-import, env, dns, every deploy-lifecycle sub, metrics):
+  `--profile` flag > `KAMPODINE_PROFILE` env > config defaultProfile >
+  the legacy env behavior. A resolved profile's host/sshKey/proxyHost feed
+  the SAME ladders those scripts already use (explicit per-invocation
+  flags always win; profile.proxyHost replaces the `PROXY_HOST` env in
+  deploy and the health header in bluegreen/status). The deploy ledger
+  already records host, so `deploy list`/ledger filtering work
+  per-profile automatically. Reads/rewrites require jq; machines without
+  a config are unaffected
+- **`kampodine metrics`** — one-shot VM snapshot over ONE ssh
+  round-trip: load + CPU count, memory (used/available from
+  /proc/meminfo), disk WITH an images/data/other breakdown (`df` for /
+  plus `du -sm` for /var/lib/containers and /data), per-container CPU/MEM
+  via `podman stats --no-stream` (busybox-safe remote commands; all
+  parsing is client-side), uptime, top-5 processes by RSS.
+  `--warn-disk <pct>` (default 90) exits 1 when root disk usage is at or
+  above the threshold — the SAME verdict logic as deploy's fail-closed
+  gate (reuses the common.sh disk helpers). `--watch N [--count M]`
+  re-snapshots every N seconds. `status --verbose` folds the SAME
+  snapshot in. NO continuous monitoring — Beszel owns that; this is the
+  CLI triage view
+- **`kampodine backup list|download|verify|restore-plan`** — OCI Object
+  Storage family (bucket `--bucket`, default esellar-libsql-backups;
+  auth via the oci CLI's config file (`--profile`, dns convention) or
+  `--instance-principal` — NEVER stored credentials). `list` renders
+  objects + sizes + timestamps and, when LIST is denied by policy, prints
+  the exact policy shape that is missing (real case: an instance
+  principal without inspect/read on list). `download` works even when
+  LIST is denied, verifies sha256 (opc-meta-sha256/content-sha256) or
+  MD5 (opc-content-md5) metadata when present, writes the output 0600
+  and prints fingerprint summaries — never contents. `verify` is
+  READ-ONLY: the local .db image travels over ssh stdin to the VM's
+  `/usr/local/bin/restore-verify` (exit codes propagate); .tgz archives
+  are extracted to a scratch dir and every .db member piped the same
+  way — it NEVER writes into /data/tenants. `restore-plan` prints the
+  documented stop/swap/start sequence bound to the object and NEVER
+  executes a mutating step (proven hermetically: ssh/oci stubs that exit
+  99 stay untouched)
+- **Host init auto-detection — groundwork for non-Alpine targets.** On
+  first connect `init_detect` probes `rc-service` (OpenRC) vs
+  `systemctl` (systemd) vs neither, and caches the verdict in the active
+  profile entry (`"init"` field; per-process without a profile). One
+  helper, `svc_action <name> <start|stop|restart>`, emits the
+  init-specific command — migrated call sites: deploy's restart path +
+  its failure diagnostics, `deploy-lifecycle restart`, and the status
+  roll call. Behavior on Alpine is byte-identical (existing tests pin
+  the exact `rc-service` strings). A host with NEITHER init refuses
+  service actions with guidance instead of guessing. NO per-OS
+  vm-prepare / distro bootstrap — detection + action resolution only
+- **Health-probe compat** — the busybox-wget health probe stays the
+  default (byte-identical), with a curl fallback for GNU-only hosts;
+  detection piggybacks the same first-connect probe
+  (`REMOTE_HTTP_CLIENT`)
+- Help everywhere held: every new command and subcommand (config init/
+  list/show/set-default/remove, metrics, backup list/download/verify/
+  restore-plan) answers `--help`/`-h` with usage + examples; the
+  top-level index gained CONFIG / METRICS / BACKUP groups; help
+  coverage, cli-dispatch, and script-gates gates extended for every new
+  script
+
+## 0.5.0 — 2026-10-08
+
+Deployment lifecycle management is now native — born from a real outage:
+the VM disk hit 100% because old sha-tagged deploy images accumulated
+silently (~1GB each), prod sqlite writes failed with disk I/O errors, and
+nothing surfaced it. Plus the owner's ask: total deployments, visible
+always.
+
+- **Deployment ledger** — every `deploy` and `deploy --rollback` appends
+  `{ts, host, sha, tag, result, duration_ms, subject}` to the local
+  append-only `~/.kampodine/deployments.jsonl` (dir 0700, file 0600);
+  shas/tags/hosts/subjects only — NEVER secrets. Failed runs are recorded
+  too (EXIT trap). Every deploy run ends with a
+  `Total deployments: N · current tag: X` footer; `status` prints the count
+- **`deploy list`** — deployment history merging VM truth (podman
+  sha-tagged images + created timestamps, the RUNNING one marked) with the
+  local ledger (last result + recorded git subject per tag); pruned tags
+  stay listed from the ledger; `Total deployments: N` footer. VM
+  unreachable degrades to a ledger-only view
+- **Disk guard** — deploy reads the VM's `df` for `/var/lib/containers`
+  before the build and after cleanup: above 90% it warns LOUD with the
+  prune hint; `--require-disk <pct>` fails closed before anything ships;
+  `status --host` shows the same reading
+- **`deploy prune [--keep N] [--dry-run]`** — removes old sha-tagged deploy
+  images on the VM, ALWAYS keeping the currently-running image, `ts-rollback`,
+  and the newest N sha-tags (default 2); refuses anything a running
+  container uses; `--dry-run` prints the exact `podman rmi` commands and
+  removes nothing
+- **Ops conveniences** (all `--host`/`--ssh-key` aware, same resolution as
+  deploy): `deploy logs [--lines N]` (podman logs --tail), `deploy restart`
+  (rc-service — deploy's own restart path, status printed after), and
+  `deploy shell` (interactive sh in the api container; `shell exec -- <cmd>`
+  for one-shot)
+- **`status` grew a VM section**: disk usage, sha-tagged image count with a
+  reclaimable-by-prune estimate, and a service roll call (kampodine-api,
+  kamal-proxy, walshipper if present)
+- Help everywhere held: every new subcommand answers `--help`/`-h` with
+  usage + examples; the top-level index gained a DEPLOY LIFECYCLE section.
+  Tests grew from 114 to 171: ledger append/read (incl. rollback entries and
+  exact key-set pins), list table + total footer, prune keep-set fixtures
+  (running/ts-rollback/newest-N never removed), disk threshold matrix +
+  fail-closed, logs/restart/shell command construction, help coverage
+
 ## 0.4.0 — 2026-10-07
 
 **Breaking: project-generic naming.** Every origin-coupled default name is

package/README.md

@@ -45,9 +45,53 @@ kampodine deploy:
   public smoke: health + build-id endpoints
 ```
 
+- **`kampodine config`** — per-instance profiles in
+  `~/.kampodine/config.json` (dir 0700, file 0600): `init --name prod
+  --host root@<ip> --ssh-key <path> [--proxy-host <h>] [--group main]
+  [--set-default]`, `list` (+ `--group` cosmetic filter, default marker,
+  sshKey shown as PATH only), `show`, `set-default`, `remove` (the last
+  profile needs `--force`). FLAT — no inheritance between profiles (the
+  schema leaves room for a future `defaults` block; nothing reads one).
+  Resolution everywhere: `--profile` flag | `KAMPODINE_PROFILE` |
+  defaultProfile | the legacy env vars — a profile's host/sshKey/proxyHost
+  feed the same ladders every subcommand already uses
+- **`kampodine metrics`** — one-shot VM snapshot over one SSH round-trip:
+  load + CPUs, memory, disk with an images/data/other breakdown,
+  per-container CPU/MEM, uptime, top-5 by RSS; `--warn-disk <pct>`
+  (default 90) exits 1 at the threshold (the deploy gate's verdict
+  logic); `--watch N` to loop. `status --verbose` folds the same
+  snapshot in. One-shot CLI view only — continuous monitoring belongs to
+  Beszel
+- **`kampodine backup`** — OCI Object Storage backup family
+  (`--bucket`, default `esellar-libsql-backups`; oci config-file
+  `--profile` or `--instance-principal` auth ONLY — never stored
+  credentials): `list` (sizes + timestamps; LIST-denied prints the
+  missing-policy guidance), `download` (GET works without LIST;
+  sha256/MD5 verification against object metadata when present; output
+  0600; fingerprint summaries, never contents), `verify` (read-only:
+  pipes the local .db — or every .db member of a .tgz, extracted to a
+  scratch dir — over ssh stdin into the VM's
+  `/usr/local/bin/restore-verify`; NEVER writes into /data/tenants), and
+  `restore-plan` (prints the documented stop/swap/start sequence; never
+  executes anything)
 - **`kampodine deploy`** — stream deploy; `--version <sha>` restreams an
   existing build; `--rollback [sha]` is an instant image-tag rollback (the
-  previous image stays on the VM for exactly this)
+  previous image stays on the VM for exactly this). Every run appends to the
+  local deployment ledger and ends with a total-deployments footer; a disk
+  guard reads the VM's `df` before the build (LOUD warning above 90%,
+  `--require-disk <pct>` fails closed)
+- **`kampodine deploy <lifecycle>`** — the deployment lifecycle as native
+  subcommands (same `--host`/`--ssh-key` resolution as deploy):
+  `list` merges the VM's sha-tagged images (running one marked) with the
+  local ledger (`~/.kampodine/deployments.jsonl` — append-only history of
+  every deploy/rollback: ts, host, sha, tag, result, duration, git subject;
+  never secrets) into a table with a `Total deployments: N` footer;
+  `prune [--keep N] [--dry-run]` reclaims VM disk by removing old
+  sha-tagged images while ALWAYS keeping the running image, `ts-rollback`,
+  and the newest N (the 2026-10-08 disk-full outage is the reason this
+  exists); `logs [--lines N]`, `restart` (rc-service, deploy's own restart
+  path), and `shell` (interactive sh, or `exec -- <cmd>`) end the
+  hand-rolled-ssh era
 - **`kampodine env`** — manage the remote app env file (`/etc/kampodine/env`,
   0600 root) without ever printing a secret: `list` shows KEY + fingerprints
   only (value length + first 2 chars), `push --file <env>` uploads over ssh
@@ -70,6 +114,11 @@ kampodine deploy:
   rollback. A guest **anchor watcher** installed by `vm-prepare` holds the
   flip's IP half — add-only and inert until a flip writes its anchor config.
   Every sub-step has its own `--help` with usage + examples
+- **`kampodine status`** — deployment count (from the ledger), live health
+  through the proxy, and — with `--host` — VM disk usage (loud warning above
+  90%), the sha-tagged image count with a reclaimable-by-prune estimate, a
+  service roll call (app, kamal-proxy, walshipper if present), plus the
+  blue/green pair view; `--verbose` adds the full metrics snapshot
 - **`kampodine vm-prepare`** — first-run bootstrap of a bare Alpine host:
   sshd hardening (fresh VMs pass unattended), busybox-wget health probes (the
   golden image ships no curl), the blue-green anchor watcher, the podman
@@ -80,9 +129,18 @@ kampodine deploy:
 - **`kampodine migrate`** — sqlite migrations over SSH
 
 **Help everywhere:** `kampodine --help` (or bare `kampodine`) prints a
-grouped command index (DEPLOY / INFRA / DNS / ENV, vercel-style); every
-command — and every bluegreen/env/dns sub-step — answers `--help` with
-usage + examples. No subcommand silently does nothing on `--help`.
+grouped command index (DEPLOY / DEPLOY LIFECYCLE / INFRA / DNS / ENV /
+CONFIG / METRICS / BACKUP, vercel-style); every command — and every
+deploy-lifecycle/bluegreen/env/dns/config/backup sub-step — answers
+`--help` with usage + examples. No subcommand silently does nothing on
+`--help`.
+
+**Host init auto-detection:** on first connect kampodine probes
+`rc-service` (Alpine/OpenRC) vs `systemctl` (systemd) and caches the
+verdict in the active profile entry; service restarts and the status roll
+call emit the right commands for the detected init (`svc_action`).
+Alpine behavior is byte-identical to always. Groundwork for non-Alpine
+targets — no per-distro bootstrap.
 
 ## Kamal parity
 
@@ -92,8 +150,12 @@ usage + examples. No subcommand silently does nothing on `--help`.
 | `kamal rollback` | `kampodine deploy --rollback [sha]` |
 | `kamal details` | `kampodine bluegreen status` |
 | `kamal proxy …` | kamal-proxy container, unchanged |
-| `kamal app exec/logs` | `ssh root@<host>` |
-| `kamal reboot / app boot` | `rc-service <app> restart` over SSH |
+| `kamal app logs` | `kampodine deploy logs` |
+| `kamal app exec` | `kampodine deploy shell` (or `deploy shell exec -- <cmd>`) |
+| (no equivalent) | `kampodine deploy list` / `deploy prune` / `deploy restart` / `status` |
+| `kamal reboot / app boot` | `rc-service <app> restart` over SSH (init-aware: systemctl on systemd hosts) |
+| kamal's per-env config files | `kampodine config` profiles (`--profile` everywhere) |
+| (no equivalent) | `kampodine metrics` / `backup list|download|verify|restore-plan` |
 
 ## Building the expected image
 
@@ -156,9 +218,10 @@ automation is on the roadmap; today the stream is the automated path.
 ## Prerequisites
 
 Deploy machine: node ≥ 20, podman, ssh key access to the target, `oci` CLI
-(for bluegreen / image-import / dns), `jq` (for dns record surgery), and
-whatever secret-resolution your env-file step uses (kampodine is agnostic;
-the reference setup uses [varlock](https://varlock.dev) + pass).
+(for bluegreen / image-import / dns / backup), `jq` (for dns record surgery
++ profile config reads), and whatever secret-resolution your env-file step
+uses (kampodine is agnostic; the reference setup uses
+[varlock](https://varlock.dev) + pass).
 
 Target: a converged Alpine + Podman + OpenRC host (see above), reachable over
 ssh as root, with kamal-proxy running.

package/cli.js

@@ -16,6 +16,9 @@ const commands = {
   migrate: "migrate.sh",
   env: "env.sh",
   dns: "dns.sh",
+  config: "config.sh",
+  metrics: "metrics.sh",
+  backup: "backup.sh",
 };
 
 // Command index, grouped vercel-style. Every command (and every sub-step of
@@ -30,10 +33,17 @@ DEPLOY
   bluegreen      reserved-IP blue/green pair: status | init | provision | flip | rollback (each sub-step has --help)
   migrate        tenant db migrations over SSH
 
+DEPLOY LIFECYCLE
+  deploy list    deployment history: VM sha-tagged images (running one marked) merged with the local ledger + total deployments count
+  deploy prune   reclaim VM disk: remove old sha-tagged images (keeps running + ts-rollback + newest N); --dry-run prints exact commands
+  deploy logs    tail the running api container's logs (--lines N)
+  deploy restart restart the api service (init-aware: rc-service on Alpine, systemctl on systemd hosts)
+  deploy shell   interactive sh in the api container (exec -- <cmd> for one-shot)
+
 INFRA
   vm-prepare     first-run bootstrap of a bare Alpine host (OpenRC + podman stack)
   image-import   golden qcow2 -> OCI custom image
-  status         live health through the proxy + the blue/green pair view
+  status         live health + deployment count + VM disk/image/service state + metrics (--verbose) + the blue/green pair view
 
 DNS
   dns            OCI DNS records (oci config-file / instance-principal auth ONLY): records | add | rm
@@ -41,6 +51,16 @@ DNS
 ENV
   env            remote app env file (/etc/kampodine/env, 0600): list | push | pull — values NEVER printed, fingerprints only
 
+CONFIG
+  config         per-instance profiles (~/.kampodine/config.json, 0700/0600): init | list | show | set-default | remove
+                 resolution everywhere: --profile flag > KAMPODINE_PROFILE env > defaultProfile > legacy env
+
+METRICS
+  metrics        one-shot VM snapshot over SSH: load, memory, disk (images/data/other), containers, top procs; --watch N; --warn-disk exits 1
+
+BACKUP
+  backup         OCI Object Storage backups (config-file / instance-principal auth ONLY): list | download | verify | restore-plan
+
 Every command supports --help with usage + examples. All further args pass through to the underlying script.
 `;
 

package/kampodine.md

@@ -74,17 +74,27 @@ no registry, no tunnel, no docker, no systemd — SSH + podman only.
       npm, with package tests (script contract gates: `bash -n` + shellcheck
       over every script; cli dispatch contract: help/version/exit codes/bin
       integrity)
+- [x] per-instance profiles (`config`) — flat, multi-instance-ready, one
+      resolution ladder everywhere (v0.6.0)
+- [x] one-shot VM metrics snapshot (`metrics`, `status --verbose`) —
+      continuous monitoring stays external (Beszel) (v0.6.0)
+- [x] OCI Object Storage backup family (`backup list/download/verify/
+      restore-plan`) with policy-denied LIST guidance and read-only
+      restore-verify over stdin (v0.6.0)
+- [x] host init auto-detection (openrc vs systemd) — action-resolution
+      groundwork for non-Alpine targets, Alpine byte-identical (v0.6.0)
 
 **In progress:**
 - [ ] reference sweep (docs → kampodine invocations)
 
 ## Roadmap
 
-- `kampodine status` — live served sha + pair view in one command
 - registry-path automation (mirror mode: push to any OCI registry,
   VM-side pull, same health-gate/cutover tail)
-- genericized defaults (env schema path, service/container names,
-  deploy-host env) driven by config
+- non-Alpine targets on top of the init auto-detection groundwork
+  (per-distro bootstrap is explicitly OUT of scope for now)
+- optional `defaults` block in config.json feeding unset profile fields
+  (the schema already leaves room; NO inheritance exists today)
 
 ## Non-goals
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "kampodine",
-  "version": "0.4.0",
+  "version": "0.6.0",
   "description": "Kamal-style deploys for Alpine + Podman hosts, built on kamal-proxy",
   "license": "AGPL-3.0",
   "type": "module",

package/scripts/backup.sh

@@ -0,0 +1,290 @@
+#!/usr/bin/env bash
+# backup.sh — OCI Object Storage backup family (libSQL db backups).
+#
+# AUTH RULE (hard, same as dns.sh): the `oci` CLI's own auth ONLY — its
+# config file (--profile = OCI CONFIG profile) or instance principal (when
+# run ON a VM). kampodine NEVER accepts, stores, or logs credential material:
+# no tokens, no key material, nothing secret-shaped in args or output.
+#
+# The default bucket is esellar-libsql-backups (--bucket overrides). NOTE:
+# --profile here is the OCI config profile (dns convention) — kampodine
+# INSTANCE profiles do not apply to bucket ops; `backup verify` (the only
+# ssh leg) resolves its VM target through the standard ladder:
+# --host flag | KAMPODINE_PROFILE | config defaultProfile | KAMPODINE_HOST.
+#
+# LIST vs GET: object LIST can be denied by policy while GET still works —
+# `list` prints policy guidance when denied; `download` needs only the name.
+#
+# Usage:
+#   kampodine backup list [--prefix <prefix>] [--bucket <name>] [--profile <oci>] [--instance-principal]
+#   kampodine backup download <object> [--out <file>] [--bucket <name>] [--profile <oci>] [--instance-principal]
+#   kampodine backup verify <local.db|.tgz> [--host <user@ip>] [--ssh-key <path>]
+#   kampodine backup restore-plan <object> [--bucket <name>]
+#
+# verify is READ-ONLY: the local db image travels over ssh stdin to the VM's
+# /usr/local/bin/restore-verify; .tgz archives are extracted to a scratch
+# dir and each db member piped the same way. It NEVER writes into
+# /data/tenants. restore-plan prints the documented stop/swap/start sequence
+# and NEVER executes a mutating step.
+set -euo pipefail
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+# shellcheck source=common.sh
+source "$HERE/common.sh"
+
+BUCKET="${KAMPODINE_BACKUP_BUCKET:-esellar-libsql-backups}"
+OCI_PROFILE_NAME="${OCI_PROFILE:-default}"
+AUTH_ARGS=()
+
+say() { printf '\033[1;34m[backup]\033[0m %s\n' "$*"; }
+die() { printf '\033[1;31m[backup] FAIL:\033[0m %s\n' "$*" >&2; exit 1; }
+usage() {
+  local code="${1:-0}"
+  printf 'Usage:\n'
+  grep '^#   kampodine backup' "$0" | sed 's/^#   //'
+  cat <<'EOF'
+
+Auth is the oci CLI's own — its config file (--profile) or instance
+principal. kampodine never accepts, stores, or logs credential material.
+Bucket: --bucket (default esellar-libsql-backups). LIST-denied by policy is
+not fatal for download/verify — GET works with just the object name; `list`
+prints the exact policy shape when denied. verify is read-only over ssh
+stdin into /usr/local/bin/restore-verify; scratch dirs only, NEVER writes
+into /data/tenants. restore-plan prints the documented stop/swap/start
+sequence and NEVER executes anything.
+
+Examples:
+  kampodine backup list --prefix tenants/
+  kampodine backup list --bucket my-backups --profile my-oci-profile
+  kampodine backup download tenants/acme/20261008T050000Z.db --out /tmp/restore.db
+  kampodine backup verify /tmp/restore.db --host root@203.0.113.10
+  kampodine backup verify ./bundle.tgz          # every .db member, read-only
+  kampodine backup restore-plan tenants/acme/20261008T050000Z.db
+EOF
+  exit "$code"
+}
+
+# --- shared helpers -------------------------------------------------------------
+
+need_oci() {
+  command -v oci >/dev/null 2>&1 \
+    || die "oci CLI not found — brew install oci-cli (https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliinstall.htm), then: oci setup config"
+}
+
+# backup_expected_digest "<oci os object head json>" -> "sha256:<hex>" |
+# "md5:<base64>" | "" (verification skipped). Prefers sha256 metadata
+# (opc-meta-sha256 / content-sha256), falls back to opc-content-md5.
+backup_expected_digest() {
+  local j="$1" v=""
+  command -v jq >/dev/null 2>&1 || { printf ''; return 0; }
+  v="$(jq -r '.data["opc-meta-sha256"] // .data["content-sha256"] // empty' <<<"$j" 2>/dev/null || true)"
+  if [[ -n "$v" ]]; then
+    printf 'sha256:%s' "$v"
+    return 0
+  fi
+  v="$(jq -r '.data["opc-content-md5"] // empty' <<<"$j" 2>/dev/null || true)"
+  if [[ -n "$v" ]]; then
+    printf 'md5:%s' "$v"
+  fi
+}
+
+# hash_file <file> <sha256|md5> -> prints the digest (sha256: hex, md5:
+# base64 — the same encoding OCI reports). Portable across macOS/Linux.
+hash_file() {
+  local f="$1" alg="$2"
+  if [[ "$alg" == "sha256" ]]; then
+    if command -v sha256sum >/dev/null 2>&1; then
+      sha256sum "$f" | awk '{print $1}'
+    elif command -v shasum >/dev/null 2>&1; then
+      shasum -a 256 "$f" | awk '{print $1}'
+    else
+      return 1
+    fi
+  else
+    if command -v openssl >/dev/null 2>&1; then
+      openssl dgst -md5 -binary "$f" | base64
+    else
+      return 1
+    fi
+  fi
+}
+
+# --- dispatch -------------------------------------------------------------------
+SUB="${1:-}"
+case "$SUB" in
+  list|download|verify|restore-plan) shift ;;
+  -h|--help|help) usage 0 ;;
+  "") usage 2 ;;
+  *)
+    printf 'kampodine backup: unknown subcommand: %s\n\n' "$SUB" >&2
+    usage 2
+    ;;
+esac
+
+BUCKET_FLAG="" PREFIX="" OUT="" OBJ=""
+OCI_PROFILE_FLAG="" INSTANCE_PRINCIPAL=0
+HOST_FLAG="" KEY_FLAG=""
+KAMPODINE_HOST="${KAMPODINE_HOST:-}"
+SSH_KEY="${KAMPODINE_SSH_KEY:-}"
+PROFILE_FLAG=""
+POSITIONAL=()
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --bucket) BUCKET_FLAG="$2"; shift 2 ;;
+    --prefix) PREFIX="$2"; shift 2 ;;
+    --out) OUT="$2"; shift 2 ;;
+    --profile) OCI_PROFILE_FLAG="$2"; shift 2 ;;
+    --instance-principal) INSTANCE_PRINCIPAL=1; shift ;;
+    --host) HOST_FLAG="$2"; KAMPODINE_HOST="$2"; shift 2 ;;
+    --ssh-key) KEY_FLAG="$2"; SSH_KEY="$2"; shift 2 ;;
+    --kampodine-profile) PROFILE_FLAG="$2"; shift 2 ;;
+    -h|--help) usage 0 ;;
+    --) shift; POSITIONAL+=("$@"); break ;;
+    *) POSITIONAL+=("$1"); shift ;;
+  esac
+done
+[[ -n "$BUCKET_FLAG" ]] && BUCKET="$BUCKET_FLAG"
+[[ -n "$OCI_PROFILE_FLAG" ]] && OCI_PROFILE_NAME="$OCI_PROFILE_FLAG"
+(( INSTANCE_PRINCIPAL == 1 )) && AUTH_ARGS=(--auth instance_principal)
+OBJ="${POSITIONAL[0]:-}"
+
+case "$SUB" in
+  list)
+    need_oci
+    PREFIX_ARGS=()
+    [[ -n "$PREFIX" ]] && PREFIX_ARGS=(--prefix "$PREFIX")
+    say "objects in bucket $BUCKET${PREFIX:+ (prefix: $PREFIX)} (profile: $OCI_PROFILE_NAME${INSTANCE_PRINCIPAL:+, instance-principal}):"
+    # NOTE: --profile stays ON the first line (script-gates static scan reads it there)
+    if ! out="$(oci os object list --all --bucket-name "$BUCKET" "${PREFIX_ARGS[@]+"${PREFIX_ARGS[@]}"}" --profile "$OCI_PROFILE_NAME" ${AUTH_ARGS[@]+"${AUTH_ARGS[@]}"} 2>&1)"; then
+      case "$out" in
+        *NotAuthorizedOrNotFound*|*NotAuthorized*|*NotAuthenticated*|*"Authorization failed"*)
+          printf '\033[1;31m[backup] FAIL:\033[0m LIST denied — this identity (profile %s / instance principal) lacks INSPECT+READ on bucket %s.\n' "$OCI_PROFILE_NAME" "$BUCKET" >&2
+          printf 'The bucket compartment needs a policy like:\n' >&2
+          printf '  Allow dynamic-group <your-dg> to read buckets in compartment <compartment>\n' >&2
+          printf '  Allow dynamic-group <your-dg> to read objects in compartment <compartment>\n' >&2
+          printf 'GET works without LIST once the object name is known:\n' >&2
+          printf '  kampodine backup download <object> --bucket %s\n' "$BUCKET" >&2
+          exit 1
+          ;;
+        *)
+          die "object list failed: $out"
+          ;;
+      esac
+    fi
+    rows="$(jq -r '.data.objects[]? | [.name, (.size | tostring), (.timeCreated // "-")] | @tsv' <<<"$out")"
+    if [[ -z "$rows" ]]; then
+      say "(no objects${PREFIX:+ with prefix $PREFIX})"
+      exit 0
+    fi
+    printf '%-56s %-12s %s\n' "NAME" "SIZE" "UPDATED"
+    while IFS=$'\t' read -r n s t; do
+      printf '%-56s %-12s %s\n' "$n" "$s" "$t"
+    done <<<"$rows"
+    ;;
+
+  download)
+    need_oci
+    [[ -n "$OBJ" ]] || die "usage: kampodine backup download <object> [--out <file>] (--help)"
+    OUT="${OUT:-$(basename "$OBJ")}"
+    head_json="$(oci os object head --bucket-name "$BUCKET" --name "$OBJ" --profile "$OCI_PROFILE_NAME" ${AUTH_ARGS[@]+"${AUTH_ARGS[@]}"} 2>/dev/null || true)"
+    expected="$(backup_expected_digest "$head_json")"
+    # 0600 FROM CREATION: umask 077 around the write; chmod after.
+    ( umask 077; oci os object get --bucket-name "$BUCKET" --name "$OBJ" --file "$OUT" --profile "$OCI_PROFILE_NAME" ${AUTH_ARGS[@]+"${AUTH_ARGS[@]}"} >/dev/null ) \
+      || die "object get failed for $OBJ (bucket $BUCKET)"
+    chmod 600 "$OUT"
+    size="$(wc -c < "$OUT" | tr -d '[:space:]')"
+    fp="$(hash_file "$OUT" sha256 || true)"
+    say "downloaded $OBJ -> $OUT (0600, $size bytes)"
+    [[ -n "$fp" ]] && say "sha256: ${fp:0:16}…"
+    if [[ -z "$expected" ]]; then
+      say "integrity: skipped (no digest metadata on the object)"
+    else
+      alg="${expected%%:*}"
+      want="${expected#*:}"
+      got="$(hash_file "$OUT" "$alg" 2>/dev/null || true)"
+      if [[ -z "$got" || "$got" != "$want" ]]; then
+        printf '\033[1;31m[backup] FAIL:\033[0m digest MISMATCH (%s)\n' "$alg" >&2
+        printf '  expected: %s\n' "$want" >&2
+        printf '  got     : %s\n' "$got" >&2
+        exit 1
+      fi
+      say "integrity: verified ($alg)"
+    fi
+    ;;
+
+  verify)
+    [[ -n "$OBJ" ]] || die "usage: kampodine backup verify <local.db|.tgz> (--help)"
+    [[ -f "$OBJ" ]] || die "no such file: $OBJ"
+    case "$OBJ" in
+      *.db|*.sqlite|*.sqlite3) VERIFY_MODE="db" ;;
+      *.tgz|*.tar.gz) VERIFY_MODE="tgz" ;;
+      *) die "backup verify takes a .db (or .sqlite/.sqlite3) or .tgz file — got: $OBJ" ;;
+    esac
+    # ssh leg resolves through the kampodine ladder: --host flag | profile
+    profile_resolve "$PROFILE_FLAG" "$HOST_FLAG" "$KEY_FLAG"
+    SSH_KEY="${KEY_FLAG:-${KAMPODINE_SSH_KEY:-}}"
+    [[ -n "$KAMPODINE_HOST" ]] || die "verify needs a VM target: --host root@<ip>, --kampodine-profile <name>, KAMPODINE_PROFILE, config defaultProfile, or KAMPODINE_HOST"
+    SSH_ARGS=(-o ConnectTimeout=10 -o BatchMode=yes)
+    [[ -n "$SSH_KEY" ]] && SSH_ARGS+=(-i "$SSH_KEY")
+    # $1 is a composed remote command — client-side expansion is the design.
+    # shellcheck disable=SC2029
+    vm() { ssh "${SSH_ARGS[@]}" "$KAMPODINE_HOST" "$1"; }
+    if [[ "$(uname -s)" == "Darwin" ]]; then
+      SSH_AUTH_SOCK="$(launchctl getenv SSH_AUTH_SOCK 2>/dev/null || true)"
+      export SSH_AUTH_SOCK
+    fi
+
+    if [[ "$VERIFY_MODE" == "db" ]]; then
+      say "piping $OBJ over ssh stdin to /usr/local/bin/restore-verify on $KAMPODINE_HOST (read-only; scratch only — NEVER writes into /data/tenants)"
+      cat "$OBJ" | vm "/usr/local/bin/restore-verify /dev/stdin"
+      say "OK — restore-verify accepted the db image"
+    else
+      scratch="$(mktemp -d "${TMPDIR:-/tmp}/kampodine-verify.XXXXXX")"
+      trap 'rm -rf "$scratch"' EXIT
+      tar -xzf "$OBJ" -C "$scratch"
+      members="$(find "$scratch" -type f \( -name '*.db' -o -name '*.sqlite' -o -name '*.sqlite3' \) | LC_ALL=C sort)"
+      [[ -n "$members" ]] || die "no .db/.sqlite members in $OBJ — nothing to verify"
+      say "piping every .db member of $OBJ over ssh stdin (read-only; scratch dir $scratch)"
+      while IFS= read -r m; do
+        member_name="${m#"$scratch"/}"
+        say "verify member: $member_name"
+        cat "$m" | vm "/usr/local/bin/restore-verify /dev/stdin" \
+          || die "restore-verify FAILED for $member_name"
+      done <<<"$members"
+      say "OK — every db member passed restore-verify"
+    fi
+    ;;
+
+  restore-plan)
+    [[ -n "$OBJ" ]] || usage 2
+    # PRINT-ONLY by design: no oci, no ssh, no mutation — the plan is
+    # documentation bound to this object, never a runner.
+    cat <<EOF
+== restore plan (print-only — kampodine NEVER executes these steps) ==
+object : $OBJ
+bucket : $BUCKET
+
+ 1. stage the backup locally (download works even when LIST is denied):
+      kampodine backup download '$OBJ' --out /tmp/restore.db
+ 2. copy to the VM scratch dir (never directly into /data/tenants):
+      scp /tmp/restore.db root@<vm>:/tmp/restore.db
+ 3. verify BEFORE touching anything (read-only, over ssh stdin):
+      kampodine backup verify /tmp/restore.db --host root@<vm>
+ 4. STOP the api so writes drain:
+      ssh root@<vm> 'rc-service kampodine-api stop'
+ 5. snapshot the current state (your rollback point):
+      ssh root@<vm> 'cp -a /data/tenants/<tenant-dir> /data/tenants/<tenant-dir>.pre-restore'
+ 6. SWAP the restored file in (permissions matter):
+      ssh root@<vm> 'install -m 640 -o root -g root /tmp/restore.db /data/tenants/<tenant-dir>/db.sqlite'
+ 7. START and smoke:
+      ssh root@<vm> 'rc-service kampodine-api start'
+      kampodine status
+
+Notes:
+  - step 3 is this family's restore-verify contract: the db image travels
+    over ssh stdin; /usr/local/bin/restore-verify checks it read-only.
+  - kampodine NEVER executes any step above — this plan documents the
+    infra runbook sequence for this object; a human runs each step.
+EOF
+    ;;
+esac

package/scripts/bluegreen.sh

@@ -71,9 +71,29 @@
 # loopback; the reserved IP is checked over :80 with the prod Host header).
 set -euo pipefail
 
+HERE="${BASH_SOURCE[0]%/*}"
+# shellcheck source=common.sh
+source "$HERE/common.sh"
+
+# --profile pre-scan (kampodine INSTANCE profile): every sub-step accepts it
+# before its own flags; flip/provision have their own flag loops, so the flag
+# is lifted here once, validated, and its proxyHost feeds the status health
+# header (APP_HOST_HEADER) when the env did not pin one. The flip's own
+# ACME hostname stays derived from the reserved IP by design.
+PROFILE_FLAG=""
+args=()
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --profile) PROFILE_FLAG="$2"; shift 2 ;;
+    *) args+=("$1"); shift ;;
+  esac
+done
+set -- ${args[@]+"${args[@]}"}
+profile_resolve "$PROFILE_FLAG" "" ""
+
 PROFILE="${OCI_PROFILE:-default}"
 COMPARTMENT_NAME="${OCI_COMPARTMENT:-kampodine}"
-APP_HOST_HEADER="${APP_HOST_HEADER:-app.example.com}"
+APP_HOST_HEADER="${PROFILE_PROXY_HOST:-${APP_HOST_HEADER:-app.example.com}}"
 REPO_ROOT="${REPO_ROOT:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)}"
 SSH_OPTS=(-o ConnectTimeout=6 -o BatchMode=yes -o StrictHostKeyChecking=accept-new)