kampodine 0.1.0 → 0.2.0

package/README.md

@@ -50,10 +50,21 @@ kampodine deploy:
   previous image stays on the VM for exactly this)
 - **`kampodine bluegreen status|init|provision|flip|rollback`** — reserved
   public IP management for a two-instance blue/green pair (zero DNS change,
-  health-gated flips, auto-rollback)
+  health-gated flips, auto-rollback). **0.2.0: live-proven end-to-end** —
+  provision via platform-launch + golden-qcow2 disk injection (~9 min to a
+  booting Alpine A1; OCI custom-image import is BIOS-pinned and A1-rejects),
+  a guest **anchor watcher** shipped by `vm-prepare` (add-only, inert until
+  a flip writes `/etc/esellar/anchor.conf`), the **ACME-first flip**
+  (~27 s: anchor → OCI assign → cert for the reserved-IP hostname →
+  served-sha verify) with a ~6 s rollback, and a strict inject gate
+  (release must match Alpine `3.x`; reboot failures die loudly)
 - **`kampodine vm-prepare`** — first-run bootstrap of a bare Alpine host
-- **`kampodine image-import`** — golden qcow2 → OCI custom image
-- **`kampodine migrate`** — sqlite migrations over SSH
+  (0.2.0: sshd-hardening self-heal — fresh VMs pass unattended; busybox-wget
+  probes — the golden image ships no curl; ships `esellar-anchor`)
+- **`kampodine image-import`** — golden qcow2 → OCI custom image (0.2.0:
+  every `oci` call honors `OCI_PROFILE`; note A1 rejects imported images —
+  use `bluegreen provision` for Ampere targets)
+- **`kampodine migrate`** — sqlite migrations overSSH
 
 ## Kamal parity
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "kampodine",
-  "version": "0.1.0",
+  "version": "0.2.0",
   "description": "Kamal-style deploys for Alpine + Podman hosts, built on kamal-proxy",
   "license": "AGPL-3.0",
   "type": "module",

package/scripts/bluegreen.sh

@@ -13,11 +13,59 @@
 # Usage:
 #   kampodine bluegreen status                  # pair view: instances, IP holder, health
 #   kampodine bluegreen init                    # create the reserved public IP (dormant, unassigned)
-#   kampodine bluegreen provision <color>       # launch the second instance from the golden image
-#   kampodine bluegreen flip --to <color>       # health-gated reserved-IP flip
-#   kampodine bluegreen rollback                # flip to the other color (same gate)
+#   kampodine bluegreen provision <color>       # launch the second instance (golden image; see below)
+#   kampodine bluegreen flip --to <color>       # ACME-first health-gated flip (see below)
+#   kampodine bluegreen rollback                # unassign to DORMANT + holder guest cleanup
+#
+# FLIP = ACME-FIRST (closes the 2026-10-07 pt2 serving-leg gap):
+#   1. health gate on the target's own IP
+#   2. anchor.conf written on the TARGET guest over ssh — the esellar-anchor
+#      watcher service (shipped by vm-prepare) configures the anchor private
+#      address within one interval; flip waits for `ip addr` to show it
+#   3. OCI assigns the reserved IP to the target's anchor (secondary private
+#      ip — the primary holds the launch-time ephemeral, 409 live-proven)
+#   4. ACME on the TARGET's kamal-proxy for the reserved-IP sslip hostname
+#      (`podman exec kamal-proxy kamal-proxy deploy --host=<raddr-dashes>
+#      .sslip.io --tls`): HTTP-01 needs the hostname to already resolve to
+#      the reserved ip AND route to the target — hence AFTER the assign, and
+#      hence NOT a deploy-time cert (a deploy before the flip cannot issue
+#      for the reserved hostname, and issuing on every deploy would burn LE
+#      rate limits for VMs that never flip). kamal-proxy has no standalone
+#      issue verb — `deploy` IS its registration+ACME path, the same
+#      invocation `kampodine deploy` uses; certs persist in the
+#      kamal-proxy-config volume. sslip.io sits on the public suffix list, so
+#      the reserved-hostname quota is per-hostname.
+#   5. verify https through the reserved ip (curl --resolve, valid cert for
+#      the sslip name + /up 200) and the served sha — only then: FLIPPED.
+#   Any post-assign failure auto-rolls back: the reserved ip goes to the
+#   other color's EXISTING anchor (lookup-ONLY — drills never mint green-side
+#   artifacts) or, when none exists, back to UNASSIGNED/dormant; the failed
+#   target's anchor.conf is removed and its anchor address deleted.
+#
+# ROLLBACK = back to dormant: holder cleanup (anchor.conf removal + address
+# delete over ssh) then the OCI unassign (`--private-ip-id ""`). For a real
+# pair where traffic must move to the other color, use `flip --to <other>`
+# instead — it runs the same ACME-first sequence there.
+#
+# provision has TWO routes:
+#   1. NATIVE  — a UEFI_64 esellar-alpine* custom image exists in the
+#      compartment: launch it directly (the golden image boots as-is).
+#   2. INJECT (provision-via-migrate) — OCI pins imported custom images to
+#      firmware=BIOS and A1/Ampere is UEFI-only (exhaustively proven,
+#      RUNBOOK §Blue-green), but green itself runs Alpine on a boot volume
+#      whose image metadata is the Ubuntu PLATFORM image: green was built by
+#      platform-image launch + disk injection. With no UEFI custom image,
+#      provision queries the template instance's LIVE image-id (never
+#      hardcoded — proven A1-launchable, since the template runs on it),
+#      launches from it with the ops ssh key, then streams the golden qcow2
+#      onto the new instance's boot disk (qemu-img convert -> gzip | ssh
+#      'gunzip | sudo dd', reboot, verify /etc/alpine-release). The instance
+#      record keeps the platform image metadata — exactly like green.
 #
 # Env: OCI_PROFILE (default esellar-api), OCI_COMPARTMENT (default esellar).
+# Injection extras: ALPINE_QCOW2 (golden disk path), OPS_SSH_PUBKEY (ops
+# public key for the platform-image launch), PLATFORM_SSH_USER (default
+# ubuntu), INJECT_PROBE_SLEEP / INJECT_PROBE_TRIES (ssh wait tuning).
 # Health checks SSH to each instance's OWN ephemeral IP (the app is checked on
 # loopback; the reserved IP is checked over :80 with the prod Host header).
 set -euo pipefail
@@ -25,6 +73,7 @@ set -euo pipefail
 PROFILE="${OCI_PROFILE:-esellar-api}"
 COMPARTMENT_NAME="${OCI_COMPARTMENT:-esellar}"
 APP_HOST_HEADER="${APP_HOST_HEADER:-84-13-128-216.sslip.io}"
+REPO_ROOT="${REPO_ROOT:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)}"
 SSH_OPTS=(-o ConnectTimeout=6 -o BatchMode=yes -o StrictHostKeyChecking=accept-new)
 
 say()  { printf '%s\n' "$*"; }
@@ -67,27 +116,244 @@ reserved_ip() {
   printf '%s' "$row"
 }
 
-# primary_private_ip_ocid <compartment> <instance_ocid>
-primary_private_ip_ocid() {
-  local comp="$1" iid="$2" vnic
-  vnic="$(oci compute vnic-attachment list -c "$comp" --profile "$PROFILE" \
-    --instance-id "$iid" --query 'data[0]."vnic-id"' --raw-output 2>/dev/null || true)"
-  [[ -n "$vnic" ]] || die "no primary VNIC on $iid"
-  oci network private-ip list -c "$comp" --profile "$PROFILE" --vnic-id "$vnic" \
-    --query 'data[0].id' --raw-output 2>/dev/null
+# primary_vnic_of <instance_ocid> -> primary VNIC ocid (vnic-attachment list
+# REQUIRES the compartment flag — reverse of private-ip list, which rejects
+# it; both pinned in the stub tests)
+primary_vnic_of() {
+  local iid="$1"
+  oci compute vnic-attachment list -c "$(compartment_ocid)" --profile "$PROFILE" \
+    --instance-id "$iid" --query 'data[0]."vnic-id"' --raw-output 2>/dev/null || true
+}
+
+# reserved_anchor_ip <vnic_ocid> -> private-ip ocid to anchor the reserved
+# public ip on. OCI allows ONE public ip per private ip: the PRIMARY private
+# ip holds the instance's launch-time EPHEMERAL public ip, so assigning the
+# reserved there is a 409 Conflict ("already has a public IP", live-proven
+# 2026-10-07). The reserved anchors on a SECONDARY private ip instead: both
+# colors keep their ephemeral address + reachability, the flip moves only
+# the reserved ip between secondaries.
+# existing_anchor_ip <vnic> -> the SECONDARY private-ip ocid or empty.
+# LOOKUP-ONLY: auto-rollback/rollback resolve holders with this — they must
+# never create artifacts on a color they are merely pointing traffic at.
+existing_anchor_ip() {
+  local vnic="$1"
+  oci network private-ip list --profile "$PROFILE" --vnic-id "$vnic" \
+    --query 'data[? "is-primary" == `false` ] | [0].id' --raw-output 2>/dev/null || true
+}
+
+# reserved_anchor_ip <vnic> -> the anchor ocid, creating it when absent (the
+# flip TARGET is the only legitimate creation site).
+reserved_anchor_ip() {
+  local vnic="$1" sec
+  sec="$(existing_anchor_ip "$vnic")"
+  if [[ "$sec" == ocid1.privateip* ]]; then
+    printf '%s' "$sec"
+    return 0
+  fi
+  oci network private-ip create --profile "$PROFILE" --vnic-id "$vnic" \
+    --display-name "esellar-reserved-anchor" \
+    --query 'data.id' --raw-output 2>/dev/null
+}
+
+# Flip-time pacing (tests run with FLIP_POLL_SLEEP=0).
+FLIP_POLL_SLEEP="${FLIP_POLL_SLEEP:-5}"
+FLIP_ADDR_TRIES="${FLIP_ADDR_TRIES:-12}"   # guest watcher pickup: 12 x 5s = 60s
+FLIP_ACME_TRIES="${FLIP_ACME_TRIES:-24}"   # LE HTTP-01: 24 x 5s = 120s
+
+# Guest anchor protocol (the esellar-anchor watcher half lives in
+# infra/alpine-host/ansible/roles/container-service/files/esellar-anchor.sh —
+# both halves of the conf path must stay in sync: /etc/esellar/anchor.conf).
+# Remote commands are composed client-side BY DESIGN (vm-prepare convention);
+# the interpolated values are OCI-API derived and regex-gated at the call
+# sites, never user input.
+
+# shellcheck disable=SC2029
+write_anchor_conf() {
+  local ip="$1" addr="$2"
+  ssh "${SSH_OPTS[@]}" "root@$ip" \
+    "umask 077; mkdir -p /etc/esellar; printf 'ANCHOR_ADDR=%s\nANCHOR_IFACE=\n' '$addr' > /etc/esellar/anchor.conf && echo ANCHOR_CONF_WRITTEN"
+}
+
+# shellcheck disable=SC2029
+anchor_addr_ready() {
+  local ip="$1" addr="$2"
+  ssh "${SSH_OPTS[@]}" "root@$ip" "ip -4 addr show | grep -qF -- '$addr'"
+}
+
+# read_anchor_conf_addr <ip> -> the guest's current ANCHOR_ADDR (with prefix)
+# or empty. Quotes stripped — the conf is shell-sourceable by the watcher.
+read_anchor_conf_addr() {
+  local ip="$1" line addr
+  line="$(ssh "${SSH_OPTS[@]}" "root@$ip" \
+    'grep -h "^ANCHOR_ADDR=" /etc/esellar/anchor.conf 2>/dev/null | head -n 1' 2>/dev/null || true)"
+  addr="${line#ANCHOR_ADDR=}"
+  addr="${addr//\"/}"
+  addr="${addr//\'/}"
+  [[ "$addr" == */* ]] && printf '%s' "$addr"
+}
+
+# delete_guest_addr <ip> <addr> — the flip tool's explicit cleanup (the
+# WATCHER is add-only by design; removal is always ours, after the conf is
+# gone so it cannot be re-added).
+# shellcheck disable=SC2029
+delete_guest_addr() {
+  local ip="$1" addr="$2"
+  ssh "${SSH_OPTS[@]}" "root@$ip" \
+    "iface=\$(ip -4 route show default 2>/dev/null | awk '{print \$5; exit}'); [ -n \"\$iface\" ] && ip addr del '$addr' dev \"\$iface\" 2>/dev/null; true" \
+    >/dev/null 2>&1 || true
+}
+
+# cleanup_target_anchor <ip> <addr_with_prefix> — conf removal + address
+# deletion. The watcher may have sourced the conf just before removal and
+# re-added the address once; retry the delete across one watcher interval.
+cleanup_target_anchor() {
+  local ip="$1" addr="$2" i
+  ssh "${SSH_OPTS[@]}" "root@$ip" "rm -f /etc/esellar/anchor.conf" >/dev/null 2>&1 || true
+  for ((i = 1; i <= 3; i++)); do
+    delete_guest_addr "$ip" "$addr"
+    anchor_addr_ready "$ip" "$addr" >/dev/null 2>&1 || return 0
+    sleep "$((FLIP_POLL_SLEEP * 2))"
+  done
+  say "warning: guest anchor address $addr still present on $ip after cleanup (watcher resurrection?) — inert once the reserved ip moves, verify before reuse" >&2
+}
+
+# flip_failure_rollback <failed_color> <other_color> <comp> <reserved_ocid> <target_ip> <target_addr>
+# Post-assign failure path: reassign the reserved ip to the other color's
+# EXISTING anchor (lookup-ONLY — auto-rollback must never mint artifacts on
+# the holder we are failing away from), else unassign to dormant; then clean
+# the failed target's guest state.
+flip_failure_rollback() {
+  local to="$1" ob="$2" comp="$3" rocid="$4" tpub="$5" taddr="$6"
+  local recovered=0 brow bnic bpip
+  say "post-assign failure — auto-rollback" >&2
+  brow="$(instance_by_color "$comp" "$ob")"
+  if [[ -n "$brow" ]]; then
+    bnic="$(primary_vnic_of "$(cut -d' ' -f1 <<<"$brow")")"
+    bpip=""
+    [[ -n "$bnic" ]] && bpip="$(existing_anchor_ip "$bnic")"
+    if [[ "$bpip" == ocid1.privateip* ]]; then
+      if oci network public-ip update --public-ip-id "$rocid" --profile "$PROFILE" \
+        --private-ip-id "$bpip" --force >/dev/null 2>&1; then
+        say "rolled back to esellar-$ob (existing anchor $bpip)" >&2
+        recovered=1
+      fi
+    fi
+  fi
+  if (( ! recovered )); then
+    if oci network public-ip update --public-ip-id "$rocid" --profile "$PROFILE" \
+      --private-ip-id "" --force --wait-for-state AVAILABLE >/dev/null 2>&1; then
+      say "reserved ip UNASSIGNED (dormant) — no existing anchor on esellar-$ob" >&2
+    else
+      cleanup_target_anchor "$tpub" "$taddr"
+      die "AUTO-ROLLBACK FAILED — reserved ip state unknown; flip manually via console: $rocid"
+    fi
+  fi
+  cleanup_target_anchor "$tpub" "$taddr"
+  say "esellar-$to cleaned (anchor.conf removed, anchor address deleted) — investigate before retrying" >&2
 }
 
-# instance_healthy <ephemeral_ip> -> ssh + loopback app check
+# instance_healthy <ephemeral_ip> -> ssh + loopback app check.
+# busybox wget, NOT curl: the golden image is deliberately minimal (ssh +
+# OpenRC + busybox) and ships no curl (2026-10-07 flip attempt 3: the gate
+# failed "unhealthy" on a healthy blue because curl is absent).
 instance_healthy() {
   local ip="$1"
   [[ -n "$ip" ]] || return 1
   ssh "${SSH_OPTS[@]}" "root@$ip" \
-    "rc-service esellar-api status >/dev/null 2>&1 && curl -sf -o /dev/null http://127.0.0.1:8080/up" 2>/dev/null
+    "rc-service esellar-api status >/dev/null 2>&1 && busybox wget -q -O /dev/null http://127.0.0.1:8080/up" 2>/dev/null
 }
 
-# reserved_ip_healthy <address> -> the internet-facing check (host-routed through kamal-proxy)
-reserved_ip_healthy() {
-  curl -sf -m 8 -o /dev/null -H "Host: $APP_HOST_HEADER" "http://$1/up"
+# instance_wait_running <iid> — poll lifecycle-state to RUNNING. The OCI CLI's
+# `instance get` has NO --wait-for-state option (3.94.1: "No such option" —
+# the 2026-10-07 drill's first provision died on exactly that), so poll.
+# Dies on the terminal-bad states; everything else (PROVISIONING, STARTING…)
+# keeps the loop going.
+instance_wait_running() {
+  local iid="$1" sleep_s="${INJECT_PROBE_SLEEP:-5}" tries="${WAIT_RUNNING_TRIES:-120}"
+  local state="" i
+  for ((i = 1; i <= tries; i++)); do
+    state="$(oci compute instance get --instance-id "$iid" --profile "$PROFILE" \
+      --query 'data."lifecycle-state"' --raw-output 2>/dev/null || true)"
+    case "$state" in
+      RUNNING) return 0 ;;
+      FAILED | TERMINATED | TERMINATING) die "esellar instance reached $state — nothing to inject, check the console" ;;
+    esac
+    sleep "$sleep_s"
+  done
+  die "instance never reached RUNNING (last state: ${state:-unknown}) after $((tries * 10))s"
+}
+
+# inject_alpine <color> <ip> <qcow2> — provision-via-migrate: stream the golden
+# Alpine disk onto a RUNNING platform-image instance's boot volume. The disk
+# write that built green: qcow2 -> raw locally, gzip | ssh 'gunzip | sudo dd'
+# over the ssh-detected boot disk (lsblk PKNAME of the / mount — never
+# hardcoded), conv=fsync, `reboot -f` (the fs it would unmount is gone), then
+# verify Alpine answers on ssh. Instance + boot-volume image metadata stay the
+# platform image — exactly like green.
+inject_alpine() {
+  local color="$1" ip="$2" qcow2="$3"
+  local probe_sleep="${INJECT_PROBE_SLEEP:-5}" probe_tries="${INJECT_PROBE_TRIES:-60}"
+  local ruser="${PLATFORM_SSH_USER:-ubuntu}" rel="" raw kh
+  # Dedicated throwaway known-hosts for the WHOLE injection phase: the same
+  # IP runs sshd on TWO different host keys (Ubuntu platform first boot ->
+  # injected Alpine first boot). accept-new takes the first, then refuses
+  # the CHANGED key after the reboot — the user's known_hosts would wedge
+  # the Alpine verify forever (2026-10-07 live drill). The scrub below
+  # clears the phase file between the two boots; the user's file is never
+  # touched.
+  kh="$(mktemp "${TMPDIR:-/tmp}/esellar-inject-kh.XXXXXX")"
+  iss() { ssh -o UserKnownHostsFile="$kh" "${SSH_OPTS[@]}" "$@"; }
+  say "inject: waiting for ssh (${ruser}@${ip}, platform-image first boot)…"
+  ssh_wait_probe() { iss "${ruser}@${ip}" true; }
+  local i
+  for ((i = 1; i <= probe_tries; i++)); do
+    ssh_wait_probe >/dev/null 2>&1 && break
+    [[ "$i" == "$probe_tries" ]] && die "ssh never came up on ${ip} (${ruser}) — check the instance console connection"
+    sleep "$probe_sleep"
+  done
+  raw="$(mktemp "${TMPDIR:-/tmp}/esellar-inject-raw.XXXXXX")"
+  say "inject: converting ${qcow2} -> raw…"
+  qemu-img convert -O raw "$qcow2" "$raw"
+  say "inject: streaming golden disk -> ${ip} boot volume (gunzip | dd, conv=fsync)…"
+  if ! gzip -c "$raw" | iss "${ruser}@${ip}" \
+    'set -eu; DISK="$(lsblk -no PKNAME "$(findmnt -n -o SOURCE /)")"; [ -n "$DISK" ] || exit 3; echo "[inject] writing /dev/$DISK"; gunzip -c | sudo dd of="/dev/$DISK" bs=4M conv=fsync status=progress'; then
+    rm -f "$raw" "$kh"
+    die "disk stream to ${ip} failed — instance left UNBOOTABLE-ish (platform image partially overwritten): terminate it, do NOT flip to ${color}"
+  fi
+  rm -f "$raw"
+  # ClientAlive keepalives on the reboot call: reboot -f kills the platform
+  # sshd WITHOUT closing the TCP session, and ConnectTimeout only bounds
+  # connection ESTABLISHMENT — the pt3 drill hung the whole provisioner on
+  # that wedged session (the guest was long up when the client gave up).
+  # ClientAliveCountMax x Interval bounds a dead session to ~15s.
+  say "inject: rebooting into the injected disk (reboot -f — the old fs is gone)…"
+  # Surface the reboot failure: `|| true` here once false-INJECTED (Phase-4
+  # rehearsal finding 2026-10-07) — the wedged Ubuntu guest kept answering
+  # ssh and the probe below accepted its banner as "Alpine boots". The
+  # reboot must SUCCEED for the inject to be real (the disk was replaced).
+  if ! iss -o ClientAliveInterval=5 -o ClientAliveCountMax=3 "${ruser}@${ip}" 'sudo reboot -f' >/dev/null 2>&1; then
+    die "inject: reboot -f FAILED on ${ip} — the injection did not take (terminate, do NOT flip to ${color})"
+  fi
+  # the injected Alpine boots a NEW host key under the SAME ip — scrub the
+  # phase file so the probes below see it as a fresh accept-new
+  ssh-keygen -R "$ip" -f "$kh" >/dev/null 2>&1 || true
+  say "inject: waiting for Alpine ssh (root@${ip})…"
+  rel=""
+  for ((i = 1; i <= probe_tries; i++)); do
+    rel="$(iss "root@${ip}" 'cat /etc/alpine-release' 2>/dev/null || true)"
+    # Verify the RELEASE STRING, not non-empty output: the Phase-4 rehearsal
+    # false-INJECTED when the un-rebooted Ubuntu guest's ssh banner satisfied
+    # a non-empty check. /etc/alpine-release only exists on a real Alpine
+    # boot and reads 3.x for every image we ship.
+    if [[ "$rel" =~ ^3\.[0-9]+\.[0-9]+ ]]; then break; fi
+    rel=""
+    sleep "$probe_sleep"
+  done
+  rm -f "$kh"
+  [[ -n "$rel" ]] || die "injection streamed but no ALPINE 3.x ssh on ${ip} after reboot (got: '${rel:-nothing}') — check the serial console; terminate, do NOT flip to ${color}"
+  say "INJECTED esellar-$color: Alpine ${rel} boots on ${ip} (instance image metadata stays the platform image — like green)"
+  say "next: kampodine vm-prepare --host root@${ip} -> kampodine deploy --host root@${ip}"
+  say "then 'bluegreen.sh flip --to ${color}' (health-gated) once its app checks green."
 }
 
 cmd="${1:-}"
@@ -140,22 +406,74 @@ case "$cmd" in
     other=green; [[ "$color" == green ]] && other=blue
     orow="$(instance_by_color "$comp" "$other")"
     [[ -n "$orow" ]] || die "esellar-$other not RUNNING — need its AD/subnet as the pair template"
-    read -r oiid oad _ <<<"$orow"
+    # instance_by_color rows are "<ocid> <ephemeral_public_ip> <ad>" — field 2
+    # is the IP, the AD is field 3 (launching with the IP as AD fails).
+    read -r oiid _ oad <<<"$orow"
     subnet="$(oci compute vnic-attachment list -c "$comp" --profile "$PROFILE" \
       --instance-id "$oiid" --query 'data[0]."subnet-id"' --raw-output)"
-    image="$(oci compute image list -c "$comp" --profile "$PROFILE" --sort-by TIMECREATED \
+
+    # Route selection. NATIVE only with a UEFI_64 esellar-alpine* custom image:
+    # OCI pins IMPORTED images to firmware=BIOS and A1 is UEFI-only, so a BIOS
+    # verdict means the import would die at launch (Shape ... is not valid for
+    # image) — skip it. Otherwise take the template's LIVE image-id from its
+    # instance record and inject the golden disk (provision-via-migrate).
+    mode="" image=""
+    custom="$(oci compute image list -c "$comp" --profile "$PROFILE" --all --sort-by TIMECREATED \
       --query "data[?\"display-name\" != null && starts_with(\"display-name\", 'esellar-alpine')] | [0].id" \
       --raw-output 2>/dev/null || true)"
-    [[ "$image" == ocid1.image* ]] || die "no esellar-alpine* custom image in $COMPARTMENT_NAME — build/import via kampodine image-import first"
-    say "launching esellar-$color: image=${image:0:60}… ad=$ad subnet=${subnet:0:60}…"
-    iid="$(oci compute instance launch -c "$comp" --profile "$PROFILE" \
-      --availability-domain "$oad" --subnet-id "$subnet" --image-id "$image" \
-      --shape VM.Standard.A1.Flex --shape-config '{"ocpus":2,"memoryInGBs":12}' \
-      --assign-public-ip true --display-name "esellar-$color" \
-      --query 'data.id' --raw-output)"
+    if [[ "$custom" == ocid1.image* ]]; then
+      fw="$(oci compute image get --image-id "$custom" --profile "$PROFILE" \
+        --query 'data."launch-options"."firmware"' --raw-output 2>/dev/null || true)"
+      if [[ "$fw" == "UEFI_64" ]]; then
+        image="$custom" mode="native"
+      else
+        say "note: newest esellar-alpine* custom image is firmware=${fw:-unknown} — A1 rejects BIOS-pinned imports, skipping to the platform-image + injection route"
+      fi
+    fi
+    if [[ -z "$image" ]]; then
+      image="$(oci compute instance get --instance-id "$oiid" --profile "$PROFILE" \
+        --query 'data."image-id"' --raw-output 2>/dev/null || true)"
+      [[ "$image" == ocid1.image* ]] || die "template instance has no resolvable image-id — cannot launch or inject"
+      mode="inject"
+      qcow2="${ALPINE_QCOW2:-${REPO_ROOT}/infra/alpine-host/build/esellar-alpine-3.22.6-aarch64.qcow2}"
+      [[ -f "$qcow2" ]] || die "golden qcow2 not found: $qcow2 (build via packer, RUNBOOK § 1, or set ALPINE_QCOW2)"
+      command -v qemu-img >/dev/null 2>&1 || die "qemu-img not found in PATH (brew install qemu) — required for the qcow2 -> raw conversion"
+      # ops ssh public key for the platform-image first boot. OPS_SSH_PUBKEY
+      # (explicit file) wins; else derive from the ssh AGENT (ssh-add -L) —
+      # the agent key is what every kampodine ssh + the golden image's baked
+      # ops key expect; ~/.ssh/id_ed25519.pub can be a stale personal key
+      # (2026-10-07 drill: launch with the stale pub = unreachable instance).
+      if [[ -n "${OPS_SSH_PUBKEY:-}" ]]; then
+        [[ -f "$OPS_SSH_PUBKEY" ]] || die "ops ssh public key not found: $OPS_SSH_PUBKEY (set OPS_SSH_PUBKEY or load the key into the agent)"
+        keyfile="$OPS_SSH_PUBKEY"
+      else
+        agent_keys="$(ssh-add -L 2>/dev/null | grep -v '\.pub$' || true)"
+        [[ -n "$agent_keys" ]] || die "no ssh key available: OPS_SSH_PUBKEY unset and ssh-add lists no keys (ssh-add ~/.ssh/id_ed25519-esellar, or set OPS_SSH_PUBKEY)"
+        keyfile="$(mktemp "${TMPDIR:-/tmp}/esellar-ops-pubkey.XXXXXX")"
+        printf '%s\n' "$agent_keys" > "$keyfile"
+        chmod 600 "$keyfile"
+      fi
+    fi
+
+    say "launching esellar-$color: image=${image:0:60}… ad=$oad subnet=${subnet:0:60}… mode=$mode"
+    # NOTE: --profile stays ON the launch line (script-gates static scan reads
+    # the invocation line, not array contents).
+    launch_args=(-c "$comp" --availability-domain "$oad" --subnet-id "$subnet"
+      --image-id "$image" --shape VM.Standard.A1.Flex --shape-config '{"ocpus":2,"memoryInGBs":12}'
+      --assign-public-ip true --display-name "esellar-$color")
+    [[ "$mode" == "inject" ]] && launch_args+=(--ssh-authorized-keys-file "$keyfile")
+    iid="$(oci compute instance launch "${launch_args[@]}" --profile "$PROFILE" --query 'data.id' --raw-output)"
     say "LAUNCHED esellar-$color: $iid"
-    say "next (RUNBOOK § start-fresh): wait RUNNING -> ssh in -> kampodine vm-prepare -> kampodine deploy --host root@<ephemeral-ip>"
-    say "then 'bluegreen.sh flip --to $color' (health-gated) once its app checks green."
+    if [[ "$mode" == "native" ]]; then
+      say "next (RUNBOOK § start-fresh): wait RUNNING -> ssh in -> kampodine vm-prepare -> kampodine deploy --host root@<ephemeral-ip>"
+      say "then 'bluegreen.sh flip --to $color' (health-gated) once its app checks green."
+      exit 0
+    fi
+    say "waiting for RUNNING (inject mode)…"
+    instance_wait_running "$iid"
+    pubip="$(instance_by_color "$comp" "$color" | cut -d' ' -f2)"
+    [[ -n "$pubip" ]] || die "no ephemeral public ip on $iid yet — re-check with 'bluegreen status' and run the injection manually"
+    inject_alpine "$color" "$pubip" "$qcow2"
     ;;
 
   flip)
@@ -168,53 +486,121 @@ case "$cmd" in
         *) die "unknown flip flag: $1" ;;
       esac
     done
-    [[ "$to" == blue || "$to" == green ]] || { say "usage: kampodine bluegreen flip --to <blue|green> [--force]"; say "(rollback = flip to the other color)"; exit 2; }
+    [[ "$to" == blue || "$to" == green ]] || { say "usage: kampodine bluegreen flip --to <blue|green> [--force]"; say "(rollback = unassign to dormant: 'kampodine bluegreen rollback')"; exit 2; }
     comp="$(compartment_ocid)"
     rp="$(reserved_ip "$comp")"
     [[ -n "$rp" ]] || die "no reserved IP (run 'kampodine bluegreen init' first)"
     read -r rocid raddr _ <<<"$rp"
+    rhost="${raddr//./-}.sslip.io"
     trow="$(instance_by_color "$comp" "$to")"
     [[ -n "$trow" ]] || die "esellar-$to is not RUNNING — nothing to flip to"
-    read -r tiid _ _ <<<"$trow"
-    tpip="$(primary_private_ip_ocid "$comp" "$tiid")"
-    if instance_healthy "$(instance_by_color "$comp" "$to" | cut -d' ' -f2)"; then
+    read -r tiid tpub _ <<<"$trow"
+    tvnic="$(primary_vnic_of "$tiid")"
+    [[ -n "$tvnic" ]] || die "no primary VNIC on esellar-$to"
+    tpip="$(reserved_anchor_ip "$tvnic")"
+    [[ "$tpip" == ocid1.privateip* ]] || die "could not resolve/create the reserved-anchor secondary private ip on esellar-$to"
+    if instance_healthy "$tpub"; then
       say "target health: esellar-$to app HEALTHY on its own IP"
     else
       (( force )) || die "esellar-$to app UNHEALTHY — refusing flip (override: --force)"
       say "target health: UNHEALTHY — flipping anyway (--force)"
     fi
-    say "flip: reserved $raddr -> esellar-$to (private-ip $tpip)"
+
+    # ACME-FIRST: HTTP-01 for the reserved-IP sslip name can only complete
+    # once the hostname resolves to the reserved ip AND routes to the target,
+    # so the guest anchor address goes FIRST, then the assign, then the cert.
+    ob=green; [[ "$to" == green ]] && ob=blue
+    say "flip[1/4]: anchor conf -> esellar-$to guest ($tpub), watcher configures the address"
+    tsubnet="$(oci compute vnic-attachment list -c "$comp" --profile "$PROFILE" \
+      --instance-id "$tiid" --query 'data[0]."subnet-id"' --raw-output 2>/dev/null || true)"
+    [[ "$tsubnet" == ocid1.subnet* ]] || die "could not resolve esellar-$to's subnet id"
+    tcidr="$(oci network subnet get --subnet-id "$tsubnet" --profile "$PROFILE" \
+      --query 'data."cidr-block"' --raw-output 2>/dev/null || true)"
+    [[ "$tcidr" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/[0-9]+$ ]] || die "could not resolve esellar-$to's subnet cidr (got: ${tcidr:-none})"
+    tanchor_addr="$(oci network private-ip get --private-ip-id "$tpip" --profile "$PROFILE" \
+      --query 'data."ip-address"' --raw-output 2>/dev/null || true)"
+    [[ "$tanchor_addr" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]] || die "could not resolve the anchor private address on esellar-$to"
+    taddr_cidr="${tanchor_addr}/${tcidr##*/}"
+    if ! write_anchor_conf "$tpub" "$taddr_cidr" >/dev/null 2>&1; then
+      die "anchor.conf write failed on esellar-$to ($tpub) — nothing mutated, flip aborted"
+    fi
+    addr_ok=0
+    for ((i = 1; i <= FLIP_ADDR_TRIES; i++)); do
+      if anchor_addr_ready "$tpub" "$taddr_cidr" >/dev/null 2>&1; then addr_ok=1; break; fi
+      sleep "$FLIP_POLL_SLEEP"
+    done
+    if (( ! addr_ok )); then
+      cleanup_target_anchor "$tpub" "$taddr_cidr"
+      die "guest watcher never configured $taddr_cidr on esellar-$to — is rc-service esellar-anchor running? (conf removed, NOTHING mutated)"
+    fi
+    say "flip[2/4]: guest answers on $taddr_cidr — reserved $raddr -> esellar-$to (anchor $tpip)"
     oci network public-ip update --public-ip-id "$rocid" --profile "$PROFILE" \
       --private-ip-id "$tpip" --force --wait-for-state ASSIGNED >/dev/null \
-      || die "OCI flip call failed — reserved IP state unchanged or unknown; run 'kampodine bluegreen status'"
-    sleep 3
-    if reserved_ip_healthy "$raddr"; then
-      say "FLIPPED: $APP_HOST_HEADER now serves from esellar-$to via $raddr (public check green)"
-    else
-      say "public check FAILED through $raddr post-flip" >&2
-      ob=green; [[ "$to" == green ]] && ob=blue
-      say "auto-rollback: moving reserved IP back to esellar-$ob" >&2
-      brow="$(instance_by_color "$comp" "$ob")"
-      if [[ -n "$brow" ]]; then
-        bpip="$(primary_private_ip_ocid "$comp" "$(cut -d' ' -f1 <<<"$brow")")"
-        oci network public-ip update --public-ip-id "$rocid" --profile "$PROFILE" \
-          --private-ip-id "$bpip" --force >/dev/null \
-          && say "rolled back to esellar-$ob — investigate esellar-$to before retrying" >&2 \
-          || die "ROLLBACK FAILED — flip manually via console: $rocid"
-      fi
+      || { cleanup_target_anchor "$tpub" "$taddr_cidr"; die "OCI flip call failed — guest conf removed; run 'kampodine bluegreen status'"; }
+    say "flip[3/4]: reserved IP ASSIGNED — registering $rhost on esellar-$to's kamal-proxy (ACME HTTP-01 through the reserved ip)…"
+    if ! ssh "${SSH_OPTS[@]}" "root@$tpub" \
+      "podman exec kamal-proxy kamal-proxy deploy esellar-api --host=$rhost --target=esellar-api:8080 --tls --health-check-path=/api/auth/ok"; then
+      flip_failure_rollback "$to" "$ob" "$comp" "$rocid" "$tpub" "$taddr_cidr"
       exit 1
     fi
+    acme_ok=0
+    for ((i = 1; i <= FLIP_ACME_TRIES; i++)); do
+      if curl -sf -m 8 --resolve "$rhost:443:$raddr" "https://$rhost/up"; then acme_ok=1; break; fi
+      sleep "$FLIP_POLL_SLEEP"
+    done
+    if (( ! acme_ok )); then
+      say "https://$rhost/ never answered with a VALID cert through $raddr (rate limits? HTTP-01 unreachable?)" >&2
+      flip_failure_rollback "$to" "$ob" "$comp" "$rocid" "$tpub" "$taddr_cidr"
+      exit 1
+    fi
+    served="$(curl -s -m 8 --resolve "$rhost:443:$raddr" "https://$rhost/api/auth/ok" || true)"
+    say "flip[4/4]: cert for $rhost VALID + serving through $raddr"
+    say "FLIPPED: https://$raddr/ (https://$rhost/) now serves from esellar-$to"
+    say "served /api/auth/ok: ${served:-<no body>}"
     ;;
 
   rollback)
+    # DORMANT rollback (2026-10-07 pt3 contract): holder guest cleanup
+    # (anchor.conf removal + anchor address delete — the flip tool's explicit
+    # job; the esellar-anchor watcher is add-only) then the OCI unassign
+    # (documented CLI semantics: an empty --private-ip-id unassigns). For a
+    # real pair where traffic must land on the other color, use
+    # 'flip --to <other>' — it runs the full ACME-first sequence there.
     comp="$(compartment_ocid)"
     rp="$(reserved_ip "$comp")"
     [[ -n "$rp" ]] || die "no reserved IP"
-    read -r _ _ rholder <<<"$rp"
-    cur=blue; [[ "$rholder" == *green* ]] || cur=green
-    other=green; [[ "$cur" == green ]] && other=blue
-    say "current holder: esellar-$cur — rolling back to esellar-$other"
-    exec "$0" flip --to "$other"
+    read -r rocid raddr rholder <<<"$rp"
+    if [[ -z "$rholder" || "$rholder" == "-" ]]; then
+      say "reserved $raddr already UNASSIGNED (dormant) — nothing to roll back"
+      exit 0
+    fi
+    # resolve the holder color by matching its anchor (LOOKUP-ONLY)
+    holder_color="" holder_ip=""
+    for color in blue green; do
+      crow="$(instance_by_color "$comp" "$color")"
+      [[ -n "$crow" ]] || continue
+      cnic="$(primary_vnic_of "$(cut -d' ' -f1 <<<"$crow")")"
+      capip=""
+      [[ -n "$cnic" ]] && capip="$(existing_anchor_ip "$cnic")"
+      if [[ "$capip" == "$rholder" ]]; then
+        holder_color="$color"
+        holder_ip="$(cut -d' ' -f2 <<<"$crow")"
+        break
+      fi
+    done
+    if [[ -z "$holder_color" ]]; then
+      say "warning: reserved $raddr held by an anchor of neither RUNNING color (terminated instance?) — unassigning without guest cleanup"
+    else
+      say "current holder: esellar-$holder_color ($holder_ip) — cleaning the guest anchor, then unassigning"
+      conf_addr="$(read_anchor_conf_addr "$holder_ip")"
+      cleanup_target_anchor "$holder_ip" "$conf_addr"
+      say "esellar-$holder_color guest cleaned (anchor.conf removed${conf_addr:+, address $conf_addr deleted})"
+    fi
+    say "unassigning reserved $raddr (-> dormant)…"
+    oci network public-ip update --public-ip-id "$rocid" --profile "$PROFILE" \
+      --private-ip-id "" --force --wait-for-state AVAILABLE >/dev/null \
+      || die "unassign failed — check 'kampodine bluegreen status' and the console"
+    say "ROLLED BACK: reserved $raddr UNASSIGNED (dormant)"
     ;;
 
   *)

package/scripts/image-import.sh

@@ -14,6 +14,8 @@
 # Auth: ~/.oci/config (brew install oci-cli; oci setup config) by default, or
 # --from-pass to source API credentials from the local pass store — the key
 # NEVER enters this repo, the image, or any log (values only go into env vars).
+# Every oci call pins --profile $PROFILE (OCI_PROFILE, default esellar-api) —
+# the CLI's DEFAULT profile is NOT the esellar tenancy.
 # Placeholder pass paths (create yours to match):
 #   esellar/oci/user         user OCID
 #   esellar/oci/tenancy      tenancy OCID
@@ -31,6 +33,7 @@ IMAGE=""
 BUCKET="${OCI_IMPORT_BUCKET:-esellar-image-import}"
 NAME_PREFIX="${OCI_IMPORT_NAME:-esellar-alpine}"
 COMPARTMENT_NAME="${OCI_COMPARTMENT:-esellar}"
+PROFILE="${OCI_PROFILE:-esellar-api}"
 FROM_PASS=0
 KEEP_OBJECT=0
 
@@ -76,9 +79,9 @@ if [[ -z "$IMAGE" ]]; then
 fi
 [[ -f "$IMAGE" ]] || die "image not found: $IMAGE"
 
-COMPARTMENT_OCID="$(oci iam compartment list --all --query "data[?name=='$COMPARTMENT_NAME'].id | [0]" --raw-output 2>/dev/null || true)"
-[[ "$COMPARTMENT_OCID" == ocid1.compartment* ]] || die "compartment '$COMPARTMENT_NAME' not found (oci iam compartment list)"
-NAMESPACE="$(oci os ns get --query data --raw-output)"
+COMPARTMENT_OCID="$(oci iam compartment list --all --profile "$PROFILE" --query "data[?name=='$COMPARTMENT_NAME'].id | [0]" --raw-output 2>/dev/null || true)"
+[[ "$COMPARTMENT_OCID" == ocid1.compartment* ]] || die "compartment '$COMPARTMENT_NAME' not found (profile $PROFILE)"
+NAMESPACE="$(oci os ns get --profile "$PROFILE" --query data --raw-output)"
 [[ -n "$NAMESPACE" ]] || die "could not resolve the tenancy namespace"
 
 STAMP="$(date +%Y%m%d-%H%M%S)"
@@ -86,11 +89,23 @@ OBJECT_NAME="${NAME_PREFIX}-${STAMP}.qcow2"
 IMAGE_NAME="${NAME_PREFIX}-$(basename "$IMAGE" .qcow2 | sed 's/^esellar-alpine-//')-${STAMP}"
 
 say "uploading $(basename "$IMAGE") -> os://$BUCKET/$OBJECT_NAME"
-oci os object put -bn "$BUCKET" --file "$IMAGE" --name "$OBJECT_NAME" --force \
+oci os object put -bn "$BUCKET" --profile "$PROFILE" --file "$IMAGE" --name "$OBJECT_NAME" --force \
   || die "object upload failed (bucket exists? oci os bucket create -bn $BUCKET -c $COMPARTMENT_OCID)"
 
-say "importing as custom image '$IMAGE_NAME' (self-supported: generic OS metadata, PARAVIRTUALIZED/UEFI)…"
-IMPORT_JSON="$(oci compute image import from-object \
+say "importing as custom image '$IMAGE_NAME' (self-supported: PARAVIRTUALIZED)…"
+# A1/Ampere firmware gate (2026-10-07 drill, empirically exhausted):
+# OCI pins imported images to launch-options firmware=BIOS. It is NOT
+# derived from the --operating-system string (a recognized aarch64 string
+# still yields BIOS) and NOT re-derivable via `compute image update` (OS
+# metadata is writable, firmware is frozen). A1 shape launch validation
+# rejects BIOS images — "Shape VM.Standard.A1.Flex is not valid for image"
+# — and NO launch-time override passes it: --launch-options firmware /
+# --launch-mode CUSTOM were both rejected. There is also no image->boot-
+# volume API (bootVolumes sourceDetails rejects type "image"). The only
+# sanctioned route to an A1-launchable custom image is CAPTURE FROM A
+# RUNNING A1 INSTANCE (compute image create --instance-id). The check
+# below warns loudly when the import lands BIOS.
+IMPORT_JSON="$(oci compute image import from-object --profile "$PROFILE" \
   -c "$COMPARTMENT_OCID" \
   --bucket-name "$BUCKET" \
   --namespace "$NAMESPACE" \
@@ -98,7 +113,7 @@ IMPORT_JSON="$(oci compute image import from-object \
   --display-name "$IMAGE_NAME" \
   --source-image-type QCOW2 \
   --operating-system "Linux" \
-  --operating-system-version "Alpine (self-supported)" \
+  --operating-system-version "Alpine 3.22.6 (self-supported)" \
   --launch-mode PARAVIRTUALIZED \
   --query 'data.id' --raw-output)" || die "import call failed"
 [[ "$IMPORT_JSON" == ocid1.image* ]] || die "unexpected import response: $IMPORT_JSON"
@@ -106,7 +121,7 @@ IMPORT_JSON="$(oci compute image import from-object \
 say "polling import -> AVAILABLE (up to 30m)…"
 STATE="PENDING_IMPORT"
 for _ in $(seq 1 120); do
-  STATE="$(oci compute image get --image-id "$IMPORT_JSON" --query 'data."lifecycle-state"' --raw-output 2>/dev/null || echo UNKNOWN)"
+  STATE="$(oci compute image get --image-id "$IMPORT_JSON" --profile "$PROFILE" --query 'data."lifecycle-state"' --raw-output 2>/dev/null || echo UNKNOWN)"
   case "$STATE" in
     AVAILABLE) break ;;
     PENDING_IMPORT|IMPORTING|UPLOADING) sleep 15 ;;
@@ -115,13 +130,29 @@ for _ in $(seq 1 120); do
 done
 [[ "$STATE" == "AVAILABLE" ]] || die "import did not become AVAILABLE in 30m (state: $STATE)"
 
+# Firmware verdict — A1/Ampere (our target shape) rejects BIOS-pinned images
+# at launch (see the import-call comment above). Loud, non-fatal: other
+# (x86) shapes launch fine from a BIOS image.
+IMPORT_FIRMWARE="$(oci compute image get --image-id "$IMPORT_JSON" --profile "$PROFILE" --query 'data."launch-options"."firmware"' --raw-output 2>/dev/null || echo UNKNOWN)"
+if [[ "$IMPORT_FIRMWARE" != "UEFI_64" ]]; then
+  say "WARN: import landed firmware=$IMPORT_FIRMWARE — VM.Standard.A1.Flex (Ampere, UEFI-only) WILL reject this image at launch."
+  say "      OCI has no import-time firmware control (drill-proven 2026-10-07): the sanctioned A1 route is"
+  say "      capture-from-instance: boot the qcow2 elsewhere, then 'oci compute image create --instance-id <running-a1>'."
+  say "      x86 shapes CAN launch this image directly. See infra/alpine-host/RUNBOOK.md § blue-green."
+fi
+
 if [[ $KEEP_OBJECT -eq 0 ]]; then
   say "deleting the staged object (image data now lives in the custom image)…"
-  oci os object delete -bn "$BUCKET" --name "$OBJECT_NAME" --force >/dev/null || say "WARN: staged object delete failed (cleanup manually)"
+  oci os object delete -bn "$BUCKET" --profile "$PROFILE" --name "$OBJECT_NAME" --force >/dev/null || say "WARN: staged object delete failed (cleanup manually)"
 fi
 
 say "IMPORTED:"
 say "  image OCID: $IMPORT_JSON"
 say "  display   : $IMAGE_NAME"
+say "  firmware  : $IMPORT_FIRMWARE"
 say "next:"
-say "  launch manually: oci compute instance launch --image-id $IMPORT_JSON --shape VM.Standard.A1.Flex … (RUNBOOK.md)"
+if [[ "$IMPORT_FIRMWARE" == "UEFI_64" ]]; then
+  say "  A1-ready. kampodine bluegreen provision <blue|green> picks this image (newest esellar-alpine*)."
+else
+  say "  A1 launch will REJECT this image (firmware $IMPORT_FIRMWARE) — see the WARN above / RUNBOOK § blue-green."
+fi

package/scripts/migrate.sh

@@ -55,17 +55,31 @@ fi
 failures=0
 migrated=0
 
+RESULTS_DIR="$(mktemp -d /tmp/migrate-results.XXXXXX)"
+trap 'rm -rf "$RESULTS_DIR"' EXIT
+
 migrate_one() {
   local db_file="$1" ns="$2"
+  local marker
+  marker="${RESULTS_DIR}/$(basename "$db_file" .db).status"
   log "migrating ${db_file#"${TENANT_DIR}"/} (ns: ${ns})"
   if (cd "$REPO_ROOT" && pnpm --filter api exec tsx scripts/libsql-migrate/migrate-db.ts --db "file:${db_file}" --ns "$ns"); then
-    migrated=$((migrated + 1))
+    printf 'ok' > "$marker"
   else
+    printf 'FAIL' > "$marker"
     printf '[migrate-all][FAIL] migration failed for %s\n' "$db_file" >&2
-    failures=$((failures + 1))
   fi
 }
 
+tally() {
+  failures=0; migrated=0
+  local m
+  for m in "$RESULTS_DIR"/*.status; do
+    [[ -e "$m" ]] || continue
+    if [[ "$(cat "$m")" = "ok" ]]; then migrated=$((migrated + 1)); else failures=$((failures + 1)); fi
+  done
+}
+
 # root.db (auth/org plane) migrates FIRST. App seam (sqld-topology.ts rootDbPath)
 # resolves it at /data/tenants/root/root.db; a legacy flat /data/tenants/root.db
 # is honored as fallback.
@@ -75,16 +89,28 @@ elif [[ -f "${TENANT_DIR}/root.db" ]]; then
   migrate_one "${TENANT_DIR}/root.db" root
 fi
 
-# then tenant files sorted.
+# then tenant files sorted — bounded-parallel by default (-j 4): per-file
+# write locks are independent (one writer per file by the store model), so
+# parallelism is safe; the failure gate stays all-or-nothing either way.
+JOBS="${MIGRATE_JOBS:-4}"
 shopt -s nullglob
 tenant_files=("$TENANT_DIR"/tenant_*.db)
 shopt -u nullglob
 if [[ ${#tenant_files[@]} -gt 0 ]]; then
   sorted_tenants=()
   while IFS= read -r f; do sorted_tenants+=("$f"); done < <(printf '%s\n' "${tenant_files[@]}" | sort)
-  for db_file in "${sorted_tenants[@]}"; do
-    migrate_one "$db_file" "$(basename "$db_file" .db)"
-  done
+  if [[ "$JOBS" -le 1 ]]; then
+    for db_file in "${sorted_tenants[@]}"; do
+      migrate_one "$db_file" "$(basename "$db_file" .db)"
+    done
+  else
+    # migrate_one is marker-file based (no shared shell state) — xargs
+    # children report through $RESULTS_DIR; tally() aggregates after.
+    export RESULTS_DIR TENANT_DIR REPO_ROOT
+    export -f migrate_one log
+    printf '%s\n' "${sorted_tenants[@]}" | xargs -P "$JOBS" -I{} bash -c 'migrate_one "$@" "$(basename "$1" .db)"' _ {}
+  fi
+  tally
 fi
 
 if [[ $failures -gt 0 ]]; then

package/scripts/vm-prepare.sh

@@ -8,7 +8,9 @@
 #
 # Alpine ships NO systemd anywhere (verified v3.22 main/community + edge; the
 # golden image boots OpenRC) — everything here is OpenRC-native:
-#   1. sanity gates (UEFI boot, OpenRC tooling, sshd hardening)
+#   1. sanity gates (UEFI boot, OpenRC tooling; sshd hardening is
+#      ENSURED — drop-in + Include + restart — then gated: the golden image
+#      predates the baked hardening, RUNBOOK §Blue-green pt2 gap)
 #   2. apk repositories: ensure the v3.22 community repo (podman lives there)
 #   3. podman stack: podman podman-docker crun catatonit netavark
 #      aardvark-dns fuse-overlayfs
@@ -16,9 +18,13 @@
 #   5. /etc/containers/registries.conf (insecure 127.0.0.1:5000; search docker.io)
 #   6. sysctl net.ipv4.ip_unprivileged_port_start=80 (persisted + applied live)
 #   7. OpenRC services /etc/init.d/esellar-api + /etc/init.d/kamal-proxy
-#      (supervise-daemon around plain `podman run`), both rc-update'd into the
+#      (supervise-daemon around plain `podman run`), rc-update'd into the
 #      default runlevel — that IS boot survival (no quadlets, no systemctl)
-#   8. kamal-proxy image pulled + service UP — the first deploy execs into it
+#   8. esellar-anchor: the blue-green reserved-ip flip's GUEST half — a
+#      busybox watcher (no container) that polls /etc/esellar/anchor.conf and
+#      `ip addr add`s the anchor address at flip time. Started+enabled on every
+#      VM, inert without the conf (flip tooling writes it over ssh).
+#   9. kamal-proxy image pulled + service UP — the first deploy execs into it
 #      to issue the fresh ACME certificate
 #
 # The esellar-api container is NOT started here: neither its image (pulled by
@@ -194,6 +200,137 @@ stop_post() {
 }
 INITD_PROXY
 
+cat > "$TMPDIR_LOCAL/esellar-anchor.sh" <<'ANCHOR_WATCHER'
+#!/bin/sh
+# KEEP IN SYNC with ansible/roles/container-service/files/esellar-anchor.sh
+# (vm-prepare rendered-content convention: vm-prepare bootstraps a fresh VM
+# BEFORE the first ansible converge; ansible owns the file afterwards).
+#
+# esellar-anchor.sh — guest half of the blue-green reserved-ip flip.
+#
+# OCI assigns the reserved PUBLIC ip to a SECONDARY private ip ("the anchor")
+# on the instance VNIC, but this image has NO oracle-cloud-agent: nothing
+# configures that private ip inside the guest, so packets to the reserved ip
+# die until the address exists on the interface (RUNBOOK §Blue-green,
+# 2026-10-07 pt2 serving-leg gap). `kampodine bluegreen flip` writes
+# /etc/esellar/anchor.conf over ssh at flip time; this watcher polls it and
+# runs `ip addr add` within one interval. The unit is enabled+started on every
+# VM by default and is INERT without the conf — a VM that never flips never
+# touches its addresses.
+#
+# CONF (written by the flip tooling, root-only dir):
+#   ANCHOR_ADDR="10.0.0.14/24"   # anchor private ip + subnet prefix (required)
+#   ANCHOR_IFACE="eth0"          # optional; empty/unset = default-route iface
+#
+# ADD-ONLY BY DESIGN: this script NEVER removes or flushes addresses (rollback
+# cleanup is the flip tool's explicit ssh job). Idempotent: a present address
+# is a no-op. Only state TRANSITIONS are logged (bounded log noise), to
+# stdout — supervise-daemon tees it to /var/log/esellar-anchor.log.
+
+CONF="/etc/esellar/anchor.conf"
+INTERVAL="${ESPELLAR_ANCHOR_INTERVAL:-5}"
+STATE="boot" # last logged transition (boot | idle | added | error)
+
+log() { printf '%s esellar-anchor: %s\n' "$(date '+%Y-%m-%dT%H:%M:%S%z')" "$*"; }
+
+# supervise-daemon SIGTERMs on stop — die cleanly with the current tick.
+trap 'exit 0' TERM INT
+
+default_iface() {
+  ip -4 route show default 2>/dev/null | awk '{print $5; exit}'
+}
+
+# Dotted-quad/prefix sanity gate before anything touches `ip addr add` — the
+# conf is operator-tooling written, but sourcing it must never turn garbage
+# into an `ip` invocation.
+addr_wellformed() {
+  case "$1" in
+    [0-9]*.[0-9]*.[0-9]*.[0-9]*/*) return 0 ;;
+    *) return 1 ;;
+  esac
+}
+
+while :; do
+  ANCHOR_ADDR=""
+  ANCHOR_IFACE=""
+  # shellcheck disable=SC1090
+  [ -f "$CONF" ] && . "$CONF"
+  if [ -n "${ANCHOR_ADDR:-}" ] && addr_wellformed "$ANCHOR_ADDR"; then
+    iface="${ANCHOR_IFACE:-$(default_iface)}"
+    if [ -n "$iface" ] && ip -4 addr show dev "$iface" 2>/dev/null | grep -qF -- "$ANCHOR_ADDR"; then
+      [ "$STATE" = added ] || { log "anchor $ANCHOR_ADDR present on $iface"; STATE=added; }
+    elif [ -n "$iface" ] && ip addr add "$ANCHOR_ADDR" dev "$iface" 2>/dev/null; then
+      log "anchor ADDED $ANCHOR_ADDR dev $iface"
+      STATE=added
+    else
+      [ "$STATE" = error ] || { log "anchor add FAILED for $ANCHOR_ADDR (iface ${iface:-none})"; STATE=error; }
+    fi
+  else
+    [ "$STATE" = idle ] || { log "no anchor conf (${CONF}) — idling"; STATE=idle; }
+  fi
+  sleep "$INTERVAL"
+done
+ANCHOR_WATCHER
+
+cat > "$TMPDIR_LOCAL/esellar-anchor" <<'INITD_ANCHOR'
+#!/sbin/openrc-run
+# Managed by packages/kampodine/scripts/vm-prepare.sh — do not hand-edit.
+# KEEP IN SYNC with ansible/roles/container-service/templates/esellar-anchor.initd.j2
+# (rendered with role defaults — see the esellar-api header for the
+# vm-prepare/ansible split).
+#
+# Guest half of the blue-green reserved-ip flip: supervise-daemon runs the
+# esellar-anchor.sh watcher, which polls /etc/esellar/anchor.conf (written by
+# `kampodine bluegreen flip` over ssh at flip time) and `ip addr add`s the
+# anchor address when it appears. Started+enabled on EVERY vm by default and
+# INERT without the conf: a VM that never flips never touches its addresses —
+# unlike the container units, there is no esellar_start_containers gate.
+# Transitions land in /var/log/esellar-anchor.log.
+
+name="esellar-anchor"
+description="Blue-green reserved-ip anchor address watcher (guest half of the flip)"
+
+supervisor=supervise-daemon
+command="/usr/local/sbin/esellar-anchor.sh"
+
+# Respawn forever: the watcher itself never exits (trap TERM/INT -> exit 0 on
+# stop), so a respawn means the script died abnormally — retry gently.
+respawn_delay=5
+respawn_max=0
+
+supervise_daemon_args="--stdout /var/log/esellar-anchor.log --stderr /var/log/esellar-anchor.log"
+
+depend() {
+	need net
+}
+INITD_ANCHOR
+
+cat > "$TMPDIR_LOCAL/sshd-hardening.conf" <<'SSHD_HARDENING'
+# KEEP IN SYNC with ansible/roles/alpine-base/files/sshd-hardening.conf
+# (byte-for-byte — ansible owns the file afterwards, rendered-content
+# convention). Ensured BEFORE the hardening gate below: the golden image
+# predates the baked drop-in (RUNBOOK §Blue-green pt2 gap, closed pt3) and
+# `sshd -T` reports passwordauthentication yes on a fresh VM.
+# Keys-only management: this VM's ssh surface is the ONLY management path
+# (OCI security lists keep 22 closed to the world; access via temporary
+# scoped rule or bastion — infra/oci/README.md model still applies).
+PasswordAuthentication no
+KbdInteractiveAuthentication no
+PermitRootLogin prohibit-password
+PermitEmptyPasswords no
+MaxAuthTries 3
+X11Forwarding no
+# Reverse tunnel support for the Mac-local registry pull path
+# (ssh -R 5000:... from kampodine deploy) rides the default AllowTcpForwarding.
+GatewayPorts no
+ClientAliveInterval 30
+ClientAliveCountMax 6
+
+# deploy registry tunnel rides a REMOTE forward (ssh -R 5000) — Alpine stock
+# sshd_config ships AllowTcpForwarding no; remote-only keeps -L clients off.
+AllowTcpForwarding yes
+SSHD_HARDENING
+
 # --- 0. connectivity ----------------------------------------------------------
 say "waiting for ssh on ${HOST}…"
 READY=0
@@ -213,8 +350,16 @@ vm '! readlink /proc/1/exe 2>/dev/null | grep -q systemd' || die "systemd is PID
 vm 'command -v openrc >/dev/null && command -v rc-service >/dev/null && command -v rc-update >/dev/null && command -v supervise-daemon >/dev/null && rc-status --servicelist >/dev/null' \
   || die "OpenRC tooling missing or not operational (openrc/rc-service/rc-update/supervise-daemon, rc-status)"
 
+say "sshd hardening: ensure drop-in + Include + restart (golden image predates the baked hardening — then the gate below holds)…"
+# shellcheck disable=SC2016
+vm 'grep -q "^Include /etc/ssh/sshd_config.d/\*.conf" /etc/ssh/sshd_config || sed -i "1i Include /etc/ssh/sshd_config.d/*.conf" /etc/ssh/sshd_config' \
+  || die "could not ensure the sshd_config Include line"
+vm 'mkdir -p /etc/ssh/sshd_config.d && chmod 700 /etc/ssh/sshd_config.d'
+scp -q "${SSH_ARGS[@]}" "$TMPDIR_LOCAL/sshd-hardening.conf" "$HOST:/etc/ssh/sshd_config.d/99-esellar-hardening.conf"
+vm 'rc-service sshd restart' || die "sshd restart failed after hardening ensure"
+
 say "gate: sshd hardening effective"
-vm 'sshd -T 2>/dev/null | grep -qi "^passwordauthentication no"' || die "sshd still allows passwords (image hardening not applied?)"
+vm 'sshd -T 2>/dev/null | grep -qi "^passwordauthentication no"' || die "sshd still allows passwords (hardening ensure failed?)"
 
 # --- 2. apk repositories + podman stack ----------------------------------------
 say "ensuring the v3.22 community repo (podman stack lives there)…"
@@ -272,6 +417,25 @@ say "installing OpenRC services esellar-api + kamal-proxy (supervise-daemon arou
 scp -q "${SSH_ARGS[@]}" "$TMPDIR_LOCAL/esellar-api" "$HOST:/etc/init.d/esellar-api"
 scp -q "${SSH_ARGS[@]}" "$TMPDIR_LOCAL/kamal-proxy" "$HOST:/etc/init.d/kamal-proxy"
 vm 'chmod 755 /etc/init.d/esellar-api /etc/init.d/kamal-proxy'
+
+# --- 4b. esellar-anchor (blue-green flip guest half — watcher, no container) ----
+# Ships on EVERY vm: started + enabled now, INERT until a flip writes
+# /etc/esellar/anchor.conf over ssh (ansible container-service owns both files
+# afterwards — drift repair keeps them in sync).
+say "installing the esellar-anchor watcher service (blue-green flip guest half)…"
+# /usr/local/sbin does NOT exist on the golden image (fresh Alpine ships no
+# /usr/local hierarchy; the alpine-base role creates it — vm-prepare runs
+# BEFORE any ansible)
+vm 'mkdir -p /usr/local/sbin' || die "mkdir /usr/local/sbin failed"
+scp -q "${SSH_ARGS[@]}" "$TMPDIR_LOCAL/esellar-anchor.sh" "$HOST:/usr/local/sbin/esellar-anchor.sh"
+scp -q "${SSH_ARGS[@]}" "$TMPDIR_LOCAL/esellar-anchor" "$HOST:/etc/init.d/esellar-anchor"
+vm 'chmod 755 /usr/local/sbin/esellar-anchor.sh /etc/init.d/esellar-anchor' || die "chmod esellar-anchor files failed"
+vm 'sh -n /usr/local/sbin/esellar-anchor.sh' || die "esellar-anchor.sh does not parse (busybox sh)"
+vm 'sh -n /etc/init.d/esellar-anchor' || die "/etc/init.d/esellar-anchor does not parse"
+vm 'rc-update show default | grep -qE "^[[:space:]]*esellar-anchor[[:space:]]*\\|" || rc-update add esellar-anchor default' \
+  || die "rc-update add esellar-anchor default failed"
+vm 'rc-service esellar-anchor start' || die "rc-service esellar-anchor start failed"
+
 for SVC in esellar-api kamal-proxy; do
   vm "rc-update show default | grep -qE '^[[:space:]]*${SVC}[[:space:]]*\\|' || rc-update add ${SVC} default" \
     || die "rc-update add $SVC default failed"
@@ -281,7 +445,7 @@ done
 # re-runs after a deploy must not bounce production.
 # shellcheck disable=SC2016
 vm_sh <<'UNIT_DRIFT' || true
-for SVC in esellar-api kamal-proxy; do
+for SVC in esellar-api kamal-proxy esellar-anchor; do
   if rc-service "$SVC" status > /dev/null 2>&1; then
     echo "RUNNING: $SVC (unit file was overwritten — rc-service $SVC restart to apply, on your call)"
   fi
@@ -304,12 +468,14 @@ done
 # --- 6. optional: pre-pull the app image over the registry tunnel ---------------
 if [[ $DO_PULL -eq 1 ]]; then
   say "pre-pulling the app image through the registry tunnel (fail BEFORE the first deploy)…"
-  vm 'curl -s --max-time 3 http://127.0.0.1:5000/v2/ >/dev/null' || {
+  # busybox wget, NOT curl: the golden image ships no curl (alpine-base
+  # installs it later; vm-prepare runs BEFORE any ansible — pt3 live)
+  vm 'busybox wget -q -O /dev/null http://127.0.0.1:5000/v2/' || {
     pkill -f "ssh.*-R 5000" 2>/dev/null || true; sleep 1
     nohup ssh -R 5000:127.0.0.1:5000 -N -o ServerAliveInterval=30 -o ExitOnForwardFailure=yes "$HOST" >/tmp/esellar-tunnel.log 2>&1 &
     sleep 3
   }
-  vm 'curl -s --max-time 3 http://127.0.0.1:5000/v2/ >/dev/null' || die "registry tunnel did not come up (/tmp/esellar-tunnel.log)"
+  vm 'busybox wget -q -O /dev/null http://127.0.0.1:5000/v2/' || die "registry tunnel did not come up (/tmp/esellar-tunnel.log)"
   vm 'podman pull --tls-verify=false 127.0.0.1:5000/esellar-api:latest' || die "app image pull failed"
 else
   say "skipping app-image pre-pull (pass --pull-images, or let the first deploy pull)"
@@ -349,6 +515,13 @@ vm 'rc-update show default | grep -qE "^[[:space:]]*esellar-api[[:space:]]*\\|"'
 vm 'rc-update show default | grep -qE "^[[:space:]]*kamal-proxy[[:space:]]*\\|"' || die "kamal-proxy not in the default runlevel"
 vm 'rc-service kamal-proxy status >/dev/null' || die "kamal-proxy service not started"
 
+say "gate: esellar-anchor watcher running + INERT (no anchor conf on a fresh VM)"
+vm 'rc-update show default | grep -qE "^[[:space:]]*esellar-anchor[[:space:]]*\\|"' || die "esellar-anchor not in the default runlevel"
+vm 'rc-service esellar-anchor status >/dev/null' || die "esellar-anchor service not started"
+vm '! test -e /etc/esellar/anchor.conf' || die "/etc/esellar/anchor.conf already exists on a fresh VM (wrong machine?)"
+vm 'grep -q "no anchor conf" /var/log/esellar-anchor.log' \
+  || die "esellar-anchor is started but never logged its idle transition (watcher loop not running?)"
+
 # --- summary ---------------------------------------------------------------------
 IP="${HOST#*@}"
 SUGGESTED_PROXY_HOST="${IP//./-}.sslip.io"