@astrofoundry/pi-astro 0.34.1 → 0.36.0

package/specialists/README.md

@@ -4,7 +4,7 @@ A specialist is a Pi subagent that is the only way to operate one area of the ho
 
 | Agent | Area | What it can do |
 |---|---|---|
-| `astro.arcane` | Arcane container platform (VM 100) and its GitOps repository | Every `arcane-cli` operation except its own `config`, `auth`, `self-update`; `update-report` (fresh check of every stack with running and latest release versions); `release-notes <stack>/<service>` (GitHub notes between the two versions); `project-status`, `deployment check|wait`; `manager-upgrade <version>` and `manager-rollback <stamp>` (the documented manager upgrade, owner approval); `host <command>`: sanitized host checks (memory, stats, logs, OOM, kernel OOM, audit, capacity, storage, Docker usage, service and queue checks) and `db-list|db-dump|db-restore` for Mealie and Paperless through user `spc-arcane` on VM 100; Compose changes of GitOps-managed projects through `git show|write|commit|push`, then `arcane-cli gitops` sync |
+| `astro.arcane` | Arcane container platform (VM 100) and its GitOps repository | Every `arcane-cli` operation except its own `config`, `auth`, `self-update`; `update-report` (fresh check of every stack with running and latest release versions); `release-notes <stack>/<service>` (GitHub notes between the two versions); `project-status`, `deployment check|wait`; `stack-update <stack>` (a stack's pending image updates through GitOps, redeploy, checks, and rollback; across a major version only with `--confirm`, owner approval; held stacks in the repository's `update-holds.json` are refused); `manager-upgrade <version>` and `manager-rollback <stamp>` (the documented manager upgrade, owner approval); `host <command>`: sanitized host checks (memory, stats, logs, OOM, kernel OOM, audit, capacity, storage, Docker usage, service and queue checks) and `db-list|db-dump|db-restore` for Mealie and Paperless through user `spc-arcane` on VM 100; Compose changes of GitOps-managed projects through `git show|write|commit|push`, then `arcane-cli gitops` sync |
 | `astro.identity` | Zitadel; Pomerium, nginx, lego on VM 104; `02-pulsar-proxmox/dmz/` in the homelab repository | Any Zitadel API call (v2, management, admin, auth) with responses redacted; `dmz status|journal|config-template|render|pomerium-restart|nginx-reload|lego-renew|deploy-config`; `git` on the homelab checkout, writes limited to `02-pulsar-proxmox/dmz/` |
 | `astro.network` | UniFi, FreeRADIUS (LXC 108), Tailscale (LXC 105) | Read UniFi sites, devices, clients, networks, zones, firewall policies, WANs; create, update, delete firewall policies; RADIUS and Tailscale checks on Pulsar |
 | `astro.dns` | Technitium primary (Synapse) and secondary (LXC 107); Cloudflare zones `37pla.net`, `monadeo.com` | Read zones and records on both resolvers, resolve, stats; create and delete zones, add, update, delete records on the primary; Cloudflare zones and record add, update, delete |
@@ -12,7 +12,7 @@ A specialist is a Pi subagent that is the only way to operate one area of the ho
 | `astro.security` | Wazuh, CrowdSec, rsyslog archive on VM 106 | Read-only: `status`, `attention`, `agents`, `alerts`, `decisions`, `bouncers`, `metrics`, `wazuh-alerts`, `wazuh-vulns`, `wazuh-log`, `remote-hosts`, `remote-log`, `journal` |
 | `astro.backup` | vzdump, the shared Restic repository (Pulsar host config, Arcane app backups), GCS offsite from LXC 102 | Status, journals, and manual runs of every layer; `restic-snapshots|stats|ls`, `restic-restore` into `/var/tmp/spc-restore` on Pulsar, `restore-clean` |
 | `astro.proxmox` | Proxmox VE on Pulsar | API token: `guests`, `guest`, `get <path>`, `tasks`; `start`, `shutdown|reboot|stop --confirm`; snapshots create, delete, rollback; `set` of CPU, memory, options, network. Pulsar key: `host-status`, `host-journal`, `updates`, `guest-exec <vmid> <status|journal|df|updates>` |
-| `astro.inference` | llama.cpp on Nexus, Hermes on LXC 101, Europa health | `nexus-status|start|stop|restart|log|disk`; `hermes-status|journal|errors|restart|api-health|version|guest-status|snapshots`, `hermes-releases`, `hermes-upgrade --confirm`; `europa-health` |
+| `astro.inference` | llama.cpp on Nexus, Europa health | `nexus-status|start|stop|restart|log|disk`; `europa-health` |
 | `astro.media` | trendarr over TMDB, OMDb, and the `media` project on VM 100 (Radarr, Sonarr, SABnzbd) | `trending`, `search`, `add-movies`, `add-series` (at most 20 IMDb IDs), `sabnzbd status|pause|resume`; always trendarr's JSON plus its warnings |
 
 Call one with `/run astro.<name> -- <task>` or through the `subagent` tool (both from the bundled `astro-subagents` extension). `/run` saves the specialist's session under `~/.pi/agent/subagent-sessions/<agent>/`; `/run astro.<name> --continue -- <task>` resumes the last one started from the same directory, so the specialist keeps what it saw. Sessions older than 7 days are deleted. Specialists may call each other once (`maxSubagentDepth: 1`). They run on `openai-codex/gpt-6-astra` with `xhigh` thinking, set by `model` and `thinking` in each `agents/<name>.md`; an agent without these fields inherits the caller's model and thinking. Each tool takes `args`, an array of strings; `["--help"]` lists the subcommands.
@@ -21,7 +21,7 @@ Call one with `/run astro.<name> -- <task>` or through the `subagent` tool (both
 
 - Credentials and CLIs belong to a dedicated macOS user `specialist` on the host. Agents run as the main user and reach the wrappers only through `sudo -n -u specialist -H /usr/local/bin/specialist-cli`, the single passwordless sudo rule. General sudo asks for a password.
 - Credentials come from Vaultwarden (`homelab` / `specialists`, one Secure Note per specialist) through `specialist-cli provision`, into `0600` files. Wrappers never call `bw`.
-- The wrappers run on a root-owned Node in `/usr/local/libexec/specialist-node/` (latest LTS from nodejs.org), not on Homebrew's, because the main user owns all of `/opt/homebrew` and could otherwise swap the interpreter that reads every secret. `/usr/local/sbin/specialist-install` (root, one passwordless sudo rule) installs that Node, the wrappers, arcane-cli, and trendarr from downloads it checks against the publisher's checksums; it refuses older versions and keeps the previous one for `rollback`. Still open: `provision` runs Homebrew's `bw` and `jq`, and `edge` runs Homebrew's `gcloud`, both as `specialist`.
+- The wrappers run on a root-owned Node in `/usr/local/libexec/specialist-node/` (latest LTS from nodejs.org), not on Homebrew's, because the main user owns all of `/opt/homebrew` and could otherwise swap the interpreter that reads every secret. `/usr/local/sbin/specialist-install` (root, one passwordless sudo rule) installs that Node, the wrappers, arcane-cli, trendarr, and the tools the wrappers run with secrets (`bw` and `jq` for `provision`, `gcloud` and its Python for `edge`) from downloads it checks against the publisher's checksums; it refuses older versions and keeps the previous one for `rollback`.
 - Arcane is reached with a Zitadel service user: a fresh token per call, exchanged for a 300 s Arcane token. Other services use API keys, a PAT, tokens, or SSH keys that the targets pin to forced commands. The VPS is reached through `gcloud compute start-iap-tunnel` with a service account that may only open IAP tunnels and read instances; its gcloud state lives in `/Users/specialist/.specialists/work/gcloud`.
 - Every call is logged in `/Users/specialist/.specialists/audit.log`.
 - The `specialist-gate` extension registers the tool only inside the matching specialist child and blocks other agents from calling the CLIs or reading `/Users/specialist`. `security-guard` carries the same blocks.
@@ -30,7 +30,7 @@ Limits: `network` writes only firewall policies on UniFi. `security` is read-onl
 
 ## Enable specialists on a host
 
-Host: Cortex (macOS on Apple silicon; Homebrew `bw`, `jq`, `gcloud-cli` with `python@3.14`, git; the specialists' Node and trendarr come from `specialist-install`). Steps 1 to 3 are one-time on the services; 4 to 8 are per host.
+Host: Cortex (macOS on Apple silicon; Homebrew git; the specialists' Node, trendarr, `bw`, `jq`, `gcloud`, and gcloud's Python come from `specialist-install`). Steps 1 to 3 are one-time on the services; 4 to 8 are per host.
 
 1. **Vaultwarden.** Account `specialists@monadeo.com` with **Can view** on collection `specialists` in organization `homelab`. Secure Notes with Hidden fields (`_B64` fields hold `base64 -i <keyfile> | tr -d '\n'`):
    - `arcane`: `ZITADEL_KEY_JSON`, `GITOPS_DEPLOY_KEY_B64`, `ZITADEL_USER_ID` (plain), `SSH_KEY_ARCANE_HOST_B64` (user `spc-arcane` on VM 100, for `host`, `manager-*`), and optionally `MEALIE_API_KEY` (only `host mealie-api` reads it).
@@ -41,7 +41,7 @@ Host: Cortex (macOS on Apple silicon; Homebrew `bw`, `jq`, `gcloud-cli` with `py
    - `security`: `SSH_KEY_OBS_B64`.
    - `backup`: `SSH_KEY_PULSAR_B64`, `SSH_KEY_ARCANE_B64`.
    - `proxmox`: `PVE_TOKEN_ID` (plain, `spc-proxmox@pve!cortex`), `PVE_TOKEN_SECRET`, `SSH_KEY_PULSAR_B64`.
-   - `inference`: `SSH_KEY_NEXUS_B64`, `SSH_KEY_HERMES_B64`, `SSH_KEY_PULSAR_B64`.
+   - `inference`: `SSH_KEY_NEXUS_B64`.
    - `media`: `TMDB_ACCESS_TOKEN` (TMDB API Read Access Token), `OMDB_API_KEY`, `RADARR_API_KEY`, `SONARR_API_KEY` (Settings, General, Security; each app has one), `SABNZBD_API_KEY` (the API key, not the NZB key, which can only add NZBs and cannot read or pause the queue).
 2. **Identities.** Zitadel (`https://auth.37pla.net/ui/console`): service user `spc-arcane` with Access Token Type JWT and a JSON key; service user `spc-identity` with a Personal Access Token and the Instance Owner role. Technitium (both resolvers): a user with Zones View/Modify/Delete, Dashboard View, DNS Client View, and an API token each. Cloudflare: an account-owned API token with `Zone:Read` and `DNS:Edit` on both zones. GCP project `frontdoor-1337`: service account `spc-edge` with `roles/iap.tunnelResourceAccessor` and `roles/compute.viewer`, JSON key. Proxmox (`ssh pulsar`): user `spc-proxmox@pve`, role `SpecialistGuest` = `VM.Audit VM.PowerMgmt VM.Snapshot VM.Snapshot.Rollback VM.Config.CPU VM.Config.Memory VM.Config.Options VM.Config.Network VM.Config.CDROM VM.GuestAgent.Audit`, ACLs `PVEAuditor` on `/` and `SpecialistGuest` on `/vms` for both the user and the token `spc-proxmox@pve!cortex` (`-privsep 1`).
 3. **Services.** Arcane (Settings, Authentication, Federated Credentials): Issuer `https://auth.37pla.net`, Audience `387825541182390275`, Subject exact = `spc-arcane` user ID, Role Admin, Global, lifetime 300. UniFi: API key (Settings, API Keys) and policies allowing Cortex `10.0.40.37` to TCP 22 on `10.0.20.10`, `10.0.10.37`, `10.0.90.10`, TCP 5380 on `10.0.40.53`, `10.0.40.54`, TCP 8006 on `10.0.10.37`, and TCP 8081 on `10.0.60.5`. GitHub: `spc-arcane-gitops` as a write deploy key on `astronaute77/arcane`, `spc-edge-homelab` and `spc-identity-homelab` as write deploy keys on `astronaute77/homelab`.
@@ -57,7 +57,10 @@ Host: Cortex (macOS on Apple silicon; Homebrew `bw`, `jq`, `gcloud-cli` with `py
    printf 'cortex ALL=(root) NOPASSWD: /usr/local/sbin/specialist-install\n' > /tmp/specialist-install.sudoers && sudo visudo -cf /tmp/specialist-install.sudoers && sudo install -m 0440 -o root -g wheel /tmp/specialist-install.sudoers /etc/sudoers.d/specialist-install && rm /tmp/specialist-install.sudoers
    sudo /usr/local/sbin/specialist-install node 24.22.0
    sudo install -m 0755 -o root -g wheel <package>/specialists/entry/specialist-cli.sh /usr/local/bin/specialist-cli
-   brew install --cask gcloud-cli
+   sudo /usr/local/sbin/specialist-install jq <version>
+   sudo /usr/local/sbin/specialist-install bw <version>
+   sudo /usr/local/sbin/specialist-install python <x.y.z+YYYYMMDD>
+   sudo /usr/local/sbin/specialist-install gcloud <version>
    ```
    `specialist-cli` runs the wrappers with the Node that `specialist-install node` put in place, so that Node comes first. Add `cortex ALL=(specialist) NOPASSWD: ALL` as `/etc/sudoers.d/specialists-setup` (mode `0440`) for the duration of steps 5 to 7, then delete it. `specialist-cli` and `specialist-install` hold the same list of services; after an update that adds one (such as `media`), install both again with the same `sudo install` lines.
 5. **Bootstrap**, as `specialist` (`sudo -u specialist -H zsh`, `umask 077`): `~/.config/specialists/client.env` with `BW_SERVER=https://vault.37pla.net`, `BW_CLIENTID`, `BW_CLIENTSECRET`; `~/.config/specialists/master-password`. Both `0600`.
@@ -70,15 +73,14 @@ Host: Cortex (macOS on Apple silicon; Homebrew `bw`, `jq`, `gcloud-cli` with `py
      "homelabRemote": "git@github.com:astronaute77/homelab.git", "homelabBranch": "main", "repoArea": "02-pulsar-proxmox/dmz", "pomeriumConfig": "02-pulsar-proxmox/dmz/config.yaml" }
    { "unifiBaseUrl": "https://10.0.10.1/proxy/network/integration", "unifiCertSha256": "<from: specialist-cli network -- unifi fingerprint>", "pulsarHost": "10.0.10.37", "pulsarUser": "root" }
    { "technitiumPrimaryUrl": "http://10.0.40.53:5380", "technitiumSecondaryUrl": "http://10.0.40.54:5380", "technitiumCatalog": "catalog.invalid" }
-   { "gcloud": "/opt/homebrew/bin/gcloud", "gcloudPython": "/opt/homebrew/opt/python@3.14/bin/python3.14",
+   { "gcloud": "/usr/local/libexec/specialist-gcloud/current/bin/gcloud", "gcloudPython": "/usr/local/libexec/specialist-python/current/bin/python3",
      "project": "frontdoor-1337", "zone": "us-east1-c", "instance": "frontdoor-1337", "serviceAccount": "spc-edge@frontdoor-1337.iam.gserviceaccount.com",
      "iapLocalPort": 22337, "sshUser": "spc-edge", "publicIp": "35.237.66.101", "probeHost": "auth.37pla.net",
      "homelabRemote": "git@github.com:astronaute77/homelab.git", "homelabBranch": "main", "repoArea": "00-frontdoor-vps", "nginxConf": "00-frontdoor-vps/nginx.conf" }
    { "obsHost": "10.0.90.10", "obsUser": "observability" }
    { "pulsarHost": "10.0.10.37", "pulsarUser": "root", "arcaneHost": "10.0.40.30", "arcaneUser": "arcane" }
    { "apiUrl": "https://10.0.10.37:8006", "certSha256": "<from: specialist-cli proxmox -- fingerprint>", "node": "pulsar", "pulsarHost": "10.0.10.37", "pulsarUser": "root" }
-   { "nexusHost": "10.0.40.38", "nexusUser": "nexus", "hermesHost": "10.0.40.31", "hermesUser": "hermes", "pulsarHost": "10.0.10.37", "pulsarUser": "root",
-     "europaHealthUrl": "http://10.0.60.5:8081/health", "hermesRepo": "NousResearch/hermes-agent" }
+   { "nexusHost": "10.0.40.38", "nexusUser": "nexus", "europaHealthUrl": "http://10.0.60.5:8081/health" }
    { "trendarrCli": "/Users/specialist/.specialists/bin/trendarr/node_modules/@astrofoundry/trendarr/dist/cli.js",
      "radarrUrl": "http://10.0.40.30:7878", "radarrQualityProfileId": 7, "radarrRootFolderPath": "/media/movies",
      "sonarrUrl": "http://10.0.40.30:8989", "sonarrQualityProfileId": 7, "sonarrRootFolderPath": "/media/tv", "sonarrMonitor": "all",
@@ -93,12 +95,17 @@ Host: Cortex (macOS on Apple silicon; Homebrew `bw`, `jq`, `gcloud-cli` with `py
    sudo /usr/local/sbin/specialist-install wrappers <pi-astro version>
    sudo /usr/local/sbin/specialist-install arcane-cli <Arcane server version>
    sudo /usr/local/sbin/specialist-install trendarr <version>
-   sudo /usr/local/sbin/specialist-install rollback node|wrappers|arcane-cli|trendarr
+   sudo /usr/local/sbin/specialist-install jq|bw|gcloud <version>
+   sudo /usr/local/sbin/specialist-install python <x.y.z+YYYYMMDD>
+   sudo /usr/local/sbin/specialist-install rollback node|wrappers|arcane-cli|trendarr|python|gcloud|bw|jq
    sudo /usr/local/sbin/specialist-install status
    ```
    - `node` accepts only an LTS release listed in `https://nodejs.org/dist/index.json`, checks the tarball against `SHASUMS256.txt`, and switches `current` only after every installed wrapper answers `--help` with it; `previous` keeps the last one.
    - `wrappers` takes the pi-astro tarball from the npm registry, checks its `sha512` integrity, copies what `install.sh` copies, writes `~/.specialists/bin/VERSION`, and restores the old files if any wrapper fails `--help`. `install.sh` stays the first-time installer (it also writes the default config files).
    - `arcane-cli` checks the release tarball against `arcane_<version>_checksums.txt`; it follows the Arcane server's version.
+   - `python` installs a python-build-standalone `install_only_stripped` build (astral-sh), checked against the release's `SHA256SUMS`; before the switch it imports `ssl` and `sqlite3` and runs the current gcloud with the new Python.
+   - `gcloud` assembles the SDK from Google's component snapshot `components-v<version>.json` (core, core-nix, gcloud-deps, and their darwin-arm parts), each tarball checked against the snapshot's sha256, because Google lists the full archive's sha256 only in the table on its versioned-archives page, for the current release; it needs the installed Python and answers `gcloud version` before the switch.
+   - `bw` takes `bw-macos-arm64-<version>.zip` from the bitwarden/clients release `cli-v<version>`, checked against the asset digest from the GitHub API; `jq` takes `jq-macos-arm64` from the jqlang/jq release, checked against its `sha256sum.txt`.
    - `trendarr` installs with the root-owned Node's npm, without install scripts, into `trendarr-<version>`.
    - A version older than the installed one is refused; `rollback` swaps back to the kept previous copy. `status` prints the installed versions as JSON.
 7. **SSH targets.** Each target gets an entry script from `entry/remote/` as `/usr/local/sbin/<entry>` (`0755`, root) and the public key in the right `authorized_keys` as
@@ -107,16 +114,15 @@ Host: Cortex (macOS on Apple silicon; Homebrew `bw`, `jq`, `gcloud-cli` with `py
    - Pulsar, root: `spc-network-pulsar-entry.sh`.
    - VM 106, user `observability`: `spc-security-obs-entry.sh` (tracked in the homelab repository, installed by `02-pulsar-proxmox/observability/deploy.sh`).
    - Frontdoor VPS: `spc-edge-frontdoor-install.sh` creates user `spc-edge` (group `systemd-journal`), installs `spc-edge-frontdoor-entry.sh` and the sudoers file `spc-edge-frontdoor-sudoers`, and writes `authorized_keys`. Copy the three files and the public key to `/tmp` with `gcloud compute scp --tunnel-through-iap`, then run it as root.
-   - Pulsar, root, three more keys: `spc-backup-pulsar-entry.sh`, `spc-proxmox-pulsar-entry.sh`, `spc-inference-pulsar-entry.sh` (tracked in `02-pulsar-proxmox/pulsar/system/`).
+   - Pulsar, root, two more keys: `spc-backup-pulsar-entry.sh`, `spc-proxmox-pulsar-entry.sh` (tracked in `02-pulsar-proxmox/pulsar/system/`).
    - VM 100, user `arcane`: `spc-backup-arcane-entry.sh` (tracked as `host/spc-backup-arcane-entry` in the `astronaute77/arcane` repository, next to the host backup scripts).
-   - VM 100, user `spc-arcane` (no docker group): `spc-arcane-host-entry.sh` hands each command to the root-owned `spc-arcane-op.py` (installed as `/usr/local/sbin/spc-arcane-op`), the only command `spc-arcane-sudoers` allows; it runs the reviewed arcane-ops probes from `/usr/local/lib/spc-arcane` and the fixed manager and database steps. `spc-arcane-install.sh` creates the user and installs all of it from `/tmp/spc-arcane`; the files are tracked under `host/` in the arcane repository, the probes under `scripts/arcane_ops/`. Add `"hostAddress": "10.0.40.30", "hostUser": "spc-arcane"` to `arcane.json`; other arcane commands do not read them.
+   - VM 100, user `spc-arcane` (no docker group): `spc-arcane-host-entry.sh` hands each command to the root-owned `spc-arcane-op.py` (installed as `/usr/local/sbin/spc-arcane-op`), the only command `spc-arcane-sudoers` allows; it runs the reviewed probes from `/usr/local/lib/spc-arcane` and the fixed manager and database steps. `spc-arcane-install.sh` creates the user and installs all of it from `/tmp/spc-arcane`; the files are tracked under `host/` in the arcane repository, the probes under `host/spc-arcane-probes/` (moved from the arcane-ops CLI, retired on 2026-10-19). Add `"hostAddress": "10.0.40.30", "hostUser": "spc-arcane"` to `arcane.json`; other arcane commands do not read them.
    - Nexus, user `nexus`: `spc-inference-nexus-entry.sh` as `/usr/local/sbin/spc-inference-nexus-entry` (tracked in `04-nexus-macstudio/system/`).
-   - LXC 101, user `hermes`: `spc-inference-hermes-entry.sh` (tracked in `02-pulsar-proxmox/hermes/system/`).
 8. **Pi**, as the main user: `~/.pi/agent/specialists.json` (below) and the specialist blocks in `~/.pi/agent/security-guard.json` (`specialist-cli`, `arcane-cli`, `-u specialist` in `operations`; `/Users/specialist/**` in `reads` and `writes`; keep the default rules next to them).
    ```json
    { "entry": "/usr/local/bin/specialist-cli", "user": "specialist",
      "specialists": [
-       { "agent": "astro.arcane", "tool": "arcane", "service": "arcane", "timeoutMs": 5400000 },
+       { "agent": "astro.arcane", "tool": "arcane", "service": "arcane", "timeoutMs": 8100000 },
        { "agent": "astro.identity", "tool": "identity", "service": "identity", "timeoutMs": 120000 },
        { "agent": "astro.network", "tool": "network", "service": "network", "timeoutMs": 120000 },
        { "agent": "astro.dns", "tool": "dns", "service": "dns", "timeoutMs": 60000 },
@@ -134,16 +140,16 @@ Check: `sudo -n -u specialist -H /usr/local/bin/specialist-cli arcane --caller t
 
 The `astro-discord` extension gives Discord users access to the specialists from a channel. It runs in a headless Pi host on Cortex (LaunchAgent `com.astrofoundry.astro-discord`, user `cortex`), never in interactive sessions.
 
-- Channels are configured one by one (`channels.<id>`): `trigger` is `always` (every message that names a specialist counts) or `mention` (only messages that start with `@Cortex` or reply to the bot); `specialists` is `*` or a list. A channel with one specialist needs no prefix: every message there is a task for it. In a channel with several, `dns list the zones` names the specialist; an addressed message without a name gets a reply asking for one. Threads inherit their parent channel's entry; unlisted channels are ignored. The bot replies at once with "<specialist> is working on your task" and shows the typing indicator until the answer replaces that reply. Reactions on your message: ⏳ running, 🔒 waiting for an owner's approval, ✅ done, ❌ failed. `help` shows the specialists you may use in that channel and the commands (`status` lists running tasks; owners in the admin channel also see the `config` commands). Each channel and specialist has one conversation: a reply to the bot's answer, or another task for that specialist within 30 minutes, continues it with everything the specialist saw before; otherwise a new one starts, and a task beginning with `new` forces that. Replied-to and forwarded messages (text, embeds, attachment names) travel with the task, so "check this" on a forwarded alert works. Image attachments on the message or on what it quotes (up to 4, 8 MiB each) are downloaded and passed to the specialist, which sees them; other file types are named only. Sessions live under `~/.config/astro-discord/sessions/<specialist>/` and are deleted after 7 days. Long answers arrive as a `.md` attachment. When the bridge restarts, open approval requests are marked cancelled and running tasks interrupted in Discord; ask again. Scheduled reports are a list in the config, one entry per report: `"reports": [{ "report": "arcane", "channelId": "<id>", "time": "06:00", "timeZone": "Europe/Madrid" }, { "report": "tools", "channelId": "<admin channel id>", "time": "06:00", "timeZone": "Europe/Madrid" }]` (the earlier single `dailyReport` key is refused; move it into the list). The Arcane report posts an "Arcane Daily Report:" message every morning: each service with an update as current to target version, from a fresh `update-report`; replying to it asks the arcane specialist with the report as context. The tools report posts "Cortex Tools Report:": every tool on Cortex with a newer version (Homebrew packages installed on request and casks, Pi, pi-astro, and through `specialist-install` the wrappers, arcane-cli at the Arcane server's version, trendarr, and the specialist Node at the latest LTS), a release-notes verdict from `astro.release-reviewer` for the tools the specialists build on, an Update button per line, and one Upgrade everything for Homebrew. Only owners can press them; nothing is installed otherwise. After an update that needs it, the bridge, herdr, or Astrogate restarts as soon as nothing runs on it; after a day of waiting, a Restart now button appears. The bridge calls `sudo -n /usr/local/sbin/specialist-install` through the sudoers rule from step 4.
+- Channels are configured one by one (`channels.<id>`): `trigger` is `always` (every message that names a specialist counts) or `mention` (only messages that start with `@Cortex` or reply to the bot); `specialists` is `*` or a list. A channel with one specialist needs no prefix: every message there is a task for it. In a channel with several, `dns list the zones` names the specialist; an addressed message without a name gets a reply asking for one. Threads inherit their parent channel's entry; unlisted channels are ignored. The bot replies at once with "<specialist> is working on your task" and shows the typing indicator until the answer replaces that reply. Reactions on your message: ⏳ running, 🔒 waiting for an owner's approval, ✅ done, ❌ failed. `help` shows the specialists you may use in that channel and the commands (`status` lists running tasks; owners in the admin channel also see the `config` commands). Each channel and specialist has one conversation: a reply to the bot's answer, or another task for that specialist within 30 minutes, continues it with everything the specialist saw before; otherwise a new one starts, and a task beginning with `new` forces that. Replied-to and forwarded messages (text, embeds, attachment names) travel with the task, so "check this" on a forwarded alert works. Image attachments on the message or on what it quotes (up to 4, 8 MiB each) are downloaded and passed to the specialist, which sees them; other file types are named only. Sessions live under `~/.config/astro-discord/sessions/<specialist>/` and are deleted after 7 days. Long answers arrive as a `.md` attachment. When the bridge restarts, open approval requests are marked cancelled and running tasks interrupted in Discord; ask again. Scheduled reports are a list in the config, one entry per report: `"reports": [{ "report": "arcane", "channelId": "<id>", "time": "06:00", "timeZone": "Europe/Madrid" }, { "report": "tools", "channelId": "<admin channel id>", "time": "06:00", "timeZone": "Europe/Madrid" }]` (the earlier single `dailyReport` key is refused; move it into the list). The Arcane report posts an "Arcane Daily Report:" message every morning from a fresh `update-report`: one section per stack with updates, each service as current to target version with the kind of change, a release-notes verdict on every major version, and an Update button per stack that only owners can press (it runs `arcane stack-update <stack>`; the button turns into the outcome and the reason arrives as a note only the clicker sees). Stacks held in `update-holds.json` at the root of the arcane GitOps repository (`{ "supabase": "coordinated release: review first" }`) show the reason and get no button. Replying to the report asks the arcane specialist with the report as context. The tools report posts "Cortex Tools Report:": every tool on Cortex with a newer version (Homebrew packages installed on request and casks, Pi, pi-astro, and through `specialist-install` the wrappers, arcane-cli at the Arcane server's version, trendarr, and the specialist Node at the latest LTS), a release-notes verdict from `astro.release-reviewer` for the tools the specialists build on, an Update button per line, and one Upgrade everything for Homebrew. Only owners can press them; nothing is installed otherwise. After an update that needs it, the bridge, herdr, or Astrogate restarts as soon as nothing runs on it; after a day of waiting, a Restart now button appears. The bridge calls `sudo -n /usr/local/sbin/specialist-install` through the sudoers rule from step 4.
 - Media: where the `media` specialist is present (`config channel <#channel> always media`), `/trending [type] [min_rating] [count]`, `/search <title> [type]`, and `/downloads <status|pause|resume|unpause>` run the wrapper without a model. Lists arrive as two messages, Movies and Series, each title with a button beside it: **Add** adds it at once and turns into Added, In library, or Not found (Retry after a failure); a greyed-out In library or No IMDb ID button needs nothing. Notes such as "downloads are paused" go only to the user who clicked. Owners and `access.media` users may use them; anyone else gets a refusal only they see, and in a channel without `media` the bot names the channels that have it. Plain messages in the channel still go to the agent, for requests by name ("add the new Dune"). At startup the bridge sets the three commands as the application's global commands (servers only).
-- Arcane manager: when the morning Arcane Daily Report shows a newer manager release, the message carries an **Upgrade Arcane to vX** button. Only an owner's click on the newest report runs `arcane manager-upgrade <version>` through the entry point (one at a time, counted as open work so no bridge restart interrupts it); the button turns into the outcome, a reply gives the result, and a successful upgrade installs the matching `arcane-cli` with `specialist-install`.
+- Arcane manager: when the morning Arcane Daily Report shows a newer manager release, the `arcane` section carries an **Upgrade Arcane to vX** button. Only an owner's click on the newest report runs `arcane manager-upgrade <version>` through the entry point (one at a time, counted as open work so no bridge restart interrupts it); the button turns into the outcome, a reply gives the result, and a successful upgrade installs the matching `arcane-cli` with `specialist-install`.
 - Access: `owners` may use every specialist and decide approvals; `access.<specialist>` lists extra users for that specialist; everyone else gets no reply. Owners change the configuration from the admin channel (`adminChannelId`) without a shell: `config show`, `config channel <#channel> <always|mention> <*|dns,edge>`, `config channel <#channel> remove`, `config access <specialist> add|remove <@user>`; the bot writes `~/.config/astro-discord/config.json` on Cortex, which is also editable by hand and re-read on every message. Deny the bot's role View Channel on channels it must never read; that is the boundary the map cannot provide.
-- Approvals: a risky call (anything with `--confirm`, plus writes such as record or zone deletes, Zitadel changes, Pomerium deploys, service restarts, and on Arcane destructive actions: `down`, deletes, prunes, image, volume, or network removal, `system upgrade`, `gitops delete`, plus `manager-upgrade`, `manager-rollback`, `host db-restore`, and the Radarr host-config reads; help lookups never ask) pauses the specialist and posts Approve/Deny buttons; only owners' clicks count; no decision within `approvalTimeoutMinutes` denies it. The specialist then reports the denial. Ambiguous tasks get a question back instead of an action.
+- Approvals: a risky call (anything with `--confirm`, plus writes such as record or zone deletes, Zitadel changes, Pomerium deploys, service restarts, and on Arcane destructive actions: `down`, deletes, prunes, image, volume, or network removal, `system upgrade`, `gitops delete`, plus `manager-upgrade`, `manager-rollback`, `host db-restore`, and the Radarr host-config reads; `stack-update` asks only across a major version, through its `--confirm`; an owner's Update click on the report needs no approval; help lookups never ask) pauses the specialist and posts Approve/Deny buttons; only owners' clicks count; no decision within `approvalTimeoutMinutes` denies it. The specialist then reports the denial. Ambiguous tasks get a question back instead of an action.
 - Setup, as `cortex`: `d=$(mktemp -d) && cp ~/.pi/agent/npm/node_modules/@astrofoundry/pi-astro/extensions/astro-discord/*.ts "$d" && node --disable-warning=ExperimentalWarning "$d/setup.ts"` (Node does not strip types inside `node_modules`, so the files are copied first) prompts for the bot token and the ids, checks each against Discord, and writes `~/.config/astro-discord/{token,config.json,run.sh}` and the LaunchAgent; it prints the `launchctl bootstrap` line. After a pi-astro update restart the bridge with `launchctl kickstart -k gui/$(id -u)/com.astrofoundry.astro-discord`. Log: `~/Library/Logs/astro-discord.log`.
 - Discord side: application `Cortex` with a bot, Message Content Intent enabled, invited with scopes `bot` and `applications.commands` and permissions View Channel, Send Messages, Attach Files, Read Message History, Add Reactions.
 
 ## Rotation and operations
 
-Rotation steps for every field live in the homelab repository, `03-cortex-macmini/ROTATION.md`. After any change in Vaultwarden run `specialist-cli provision`. Host state and the sudo rules are documented in `03-cortex-macmini/README.md`; the forced-command scripts are tracked under `02-pulsar-proxmox/dmz/system/`, `02-pulsar-proxmox/pulsar/system/`, `02-pulsar-proxmox/observability/system/`, `02-pulsar-proxmox/hermes/system/`, `host/` of the `astronaute77/arcane` repository, `04-nexus-macstudio/system/`, and `00-frontdoor-vps/system/`.
+Rotation steps for every field live in the homelab repository, `03-cortex-macmini/ROTATION.md`. After any change in Vaultwarden run `specialist-cli provision`. Host state and the sudo rules are documented in `03-cortex-macmini/README.md`; the forced-command scripts are tracked under `02-pulsar-proxmox/dmz/system/`, `02-pulsar-proxmox/pulsar/system/`, `02-pulsar-proxmox/observability/system/`, `host/` of the `astronaute77/arcane` repository, `04-nexus-macstudio/system/`, and `00-frontdoor-vps/system/`.
 
 All nine planned specialists exist. See `AGENTS.md` for how to add one.

package/specialists/arcane/run.ts

@@ -1,5 +1,5 @@
 import { createHash } from "node:crypto";
-import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
+import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
 import { tmpdir } from "node:os";
 import { dirname, join } from "node:path";
 import { buildHostRemote, compareAudit, HOST_COMMANDS, type AuditFile } from "../lib/arcane-host.ts";
@@ -7,14 +7,16 @@ import { failedChecks, MANAGER_COMPOSE, MANAGER_IMAGE, normalizeVersion, olderTh
 import { readConfig } from "../lib/config.ts";
 import { containerFacts, deploymentChecks, listRows, parseDeploymentArgs, syncFacts, type DeploymentExpectation } from "../lib/deployment.ts";
 import { ServiceError, UsageError } from "../lib/errors.ts";
+import { withLock } from "../lib/lock.ts";
 import { main } from "../lib/main.ts";
-import { buildUpdateReport, parseTag, resolveRef, stackServices, type CheckEntry } from "../lib/image-updates.ts";
+import { buildUpdateReport, parseTag, resolveRef, stackServices, type CheckEntry, type UpdateRow } from "../lib/image-updates.ts";
 import { printJson, printRaw } from "../lib/output.ts";
 import { specialistsHome } from "../lib/paths.ts";
 import { run, type RunResult } from "../lib/proc.ts";
 import { HttpRegistry, parseImageRef } from "../lib/registry.ts";
 import { releaseNotesFor } from "../lib/releases.ts";
-import { ensureCheckout, git, REPO_HELP, repoCommand, repoStatus, type RepoSpec } from "../lib/repo.ts";
+import { ensureCheckout, git, REPO_HELP, repoCommand, repoStatus, safeRepoPath, type RepoSpec } from "../lib/repo.ts";
+import { commitMessage, containersProblem, crossesMajor, HOLDS_FILE, nextImageRef, parseHolds, pendingRows, replaceImageRef, type ServiceResult, type StackContainer, type StackStatus } from "../lib/stack-update.ts";
 import { ARCANE_DENIED_KEYS, filterJsonText, stripKeys } from "../lib/sanitize.ts";
 import { readSecret, secretPath } from "../lib/secrets.ts";
 import { sshFixed } from "../lib/ssh.ts";
@@ -99,8 +101,15 @@ ${Object.entries(HOST_COMMANDS)
     .map(([, c]) => `      ${c.help}`)
     .join("\n")}
   update-report [--all]       fresh image update check for every stack, with running and latest release versions
-                              from the registries; JSON with a ready table (--all also lists up-to-date services).
-                              Checks only: never pulls images or redeploys.
+                              from the registries; JSON with a ready table (--all also lists up-to-date services) and
+                              the held stacks of ${HOLDS_FILE}. Checks only: never pulls images or redeploys.
+  stack-update <stack> [--confirm]
+                              apply every pending update of one stack: pinned tags and digests move in the stack's
+                              Compose file (commit, push, wait for the sync), then the stack is redeployed with a pull
+                              and checked (containers running and healthy, images current). A failed check after a
+                              Compose change reverts it and redeploys; a pull alone cannot be undone. Refuses held
+                              stacks, stopped stacks, and arcane (manager-upgrade). An update across a major version
+                              needs --confirm, which asks an owner for approval.
   release-notes <stack>[/<service>]
                               GitHub release notes between the running and the latest version of one service's image,
                               found through the image's org.opencontainers.image.source label
@@ -160,14 +169,14 @@ function writeCallConfig(config: ArcaneConfig, token: string): string {
     return file;
 }
 
-type Cli = (args: string[]) => Promise<{ stdout: string; stderr: string; code: number }>;
+type Cli = (args: string[], timeoutMs?: number) => Promise<{ stdout: string; stderr: string; code: number }>;
 
 /** One federated token and one private config copy for every arcane-cli call of this invocation. */
 async function withCli<T>(config: ArcaneConfig, fn: (cli: Cli) => Promise<T>): Promise<T> {
     const token = await federatedToken(config);
     const callConfig = writeCallConfig(config, token);
     try {
-        return await fn((args) => run(config.arcaneCli, ["--config", callConfig, "--no-color", ...args], { env: { ARCANE_TOKEN: token }, timeoutMs: TIMEOUT_MS }));
+        return await fn((args, timeoutMs = TIMEOUT_MS) => run(config.arcaneCli, ["--config", callConfig, "--no-color", ...args], { env: { ARCANE_TOKEN: token }, timeoutMs }));
     } finally {
         rmSync(dirname(callConfig), { recursive: true, force: true });
     }
@@ -202,14 +211,30 @@ async function updateReport(config: ArcaneConfig, args: string[]): Promise<numbe
     const extra = args.filter((a) => a !== "--all");
     if (extra.length > 0) throw new UsageError(`update-report takes only --all, got: ${extra.join(" ")}`);
     const startedAt = Date.now();
+    const holds = await holdsFromOrigin(config);
     return withCli(config, async (cli) => {
         const check = await cliJson(cli, ["images", "updates", "check", "--all"]);
         const projects = await cliJson(cli, ["projects", "list", "--all"]);
-        printJson(await buildUpdateReport({ check, projects, registry: new HttpRegistry(), startedAt, all: args.includes("--all") }));
+        const report = await buildUpdateReport({ check, projects, registry: new HttpRegistry(), startedAt, all: args.includes("--all") });
+        printJson({ ...report, holds: holds.holds, ...(holds.error ? { holdsError: holds.error } : {}) });
         return 0;
     });
 }
 
+/** The held stacks as pushed to the GitOps repository, read without touching the checkout's working tree. */
+async function holdsFromOrigin(config: ArcaneConfig): Promise<{ holds: Record<string, string>; error?: string }> {
+    try {
+        const spec = repoSpec(config);
+        const dir = await ensureCheckout(spec);
+        await git(spec, dir, ["fetch", "--quiet", "origin", config.gitopsBranch]);
+        const listed = (await git(spec, dir, ["ls-tree", "--name-only", `origin/${config.gitopsBranch}`, HOLDS_FILE])).trim();
+        if (listed !== HOLDS_FILE) return { holds: {} };
+        return { holds: parseHolds(await git(spec, dir, ["show", `origin/${config.gitopsBranch}:${HOLDS_FILE}`])) };
+    } catch (err) {
+        return { holds: {}, error: `${HOLDS_FILE} unreadable: ${err instanceof Error ? err.message.slice(0, 200) : String(err)}` };
+    }
+}
+
 async function releaseNotes(config: ArcaneConfig, args: string[]): Promise<number> {
     if (args.length !== 1 || !/^[A-Za-z0-9_.-]+(\/[A-Za-z0-9_.-]+)?$/.test(args[0])) throw new UsageError("release-notes <stack>[/<service>]");
     const [stack, service] = args[0].split("/");
@@ -370,6 +395,16 @@ const SYNC_WAIT_MS = 30 * MINUTE;
  * stopped (nothing was restarted). The exit code is 0 in all three so the
  * caller always gets the answer; usage and precondition errors still fail.
  */
+/**
+ * Everything that writes the arcane-gitops checkout takes turns: an update holds it
+ * for up to about two hours (sync and redeploy waits), so the lock outlives that.
+ */
+const CHECKOUT_STALE_MS = 3 * 60 * 60_000;
+
+function checkoutLocked<T>(fn: () => Promise<T>): Promise<T> {
+    return withLock("arcane-gitops", fn, { staleMs: CHECKOUT_STALE_MS });
+}
+
 async function managerUpgrade(config: ArcaneConfig, args: string[]): Promise<number> {
     if (args.length !== 1) throw new UsageError("manager-upgrade <version>");
     const version = normalizeVersion(args[0]);
@@ -480,20 +515,198 @@ async function managerRollback(args: string[]): Promise<number> {
     return 0;
 }
 
+// ---------------------------------------------------------------- stack updates
+
+/** How long a stack's sync may take to report success at a new commit; syncs run every few minutes. */
+const STACK_SYNC_WAIT_MS = 20 * MINUTE;
+/** arcane-cli allows a project redeploy up to 30 minutes for large pulls. */
+const REDEPLOY_TIMEOUT_MS = 35 * MINUTE;
+/** Containers with a health check get this long to leave "starting". */
+const HEALTH_WAIT_MS = 5 * MINUTE;
+
+interface StackState {
+    rows: UpdateRow[];
+    services: number;
+    stopped: boolean;
+    composePath: string | null;
+}
+
+/** The stack's services with a fresh check of each of its images, resolved the way update-report resolves them. */
+async function stackState(cli: Cli, stack: string, startedAt: number): Promise<StackState> {
+    const projects = await cliJson(cli, ["projects", "list", "--all"]);
+    const services = stackServices(projects).filter((s) => s.stack === stack);
+    if (services.length === 0) throw new UsageError(`no stack named ${stack}`);
+    const project = listRows(projects).find((p) => p.name === stack);
+    const sync = listRows(await cliJson(cli, ["gitops", "list", "--all"])).filter((s) => project !== undefined && s.projectId === project.id);
+    const composePath = sync.length === 1 && typeof sync[0].composePath === "string" ? sync[0].composePath : null;
+    const registry = new HttpRegistry();
+    const resolved = new Map<string, Awaited<ReturnType<typeof resolveRef>>>();
+    for (const ref of [...new Set(services.map((s) => s.ref).filter((r): r is string => r !== null))]) {
+        const entry = parseImageRef(ref).digest ? undefined : ((await cliJson(cli, ["images", "updates", "check", ref])) as CheckEntry);
+        resolved.set(ref, await resolveRef(ref, entry, registry, startedAt));
+    }
+    const rows: UpdateRow[] = services.map((s) => {
+        const r = s.ref === null ? undefined : resolved.get(s.ref);
+        if (s.ref === null || !r) return { stack, service: s.service, ref: s.ref ?? "", image: "?", kind: "floating", running: "?", latest: "?", status: "unknown", action: "", note: "Arcane lists no image for this service" };
+        return { stack, service: s.service, ref: s.ref, image: s.ref, kind: r.kind, running: r.running, latest: r.latest, status: r.status, action: r.action, note: r.note, ...(r.latestDigest ? { latestDigest: r.latestDigest } : {}) };
+    });
+    return { rows, services: services.length, stopped: services.some((s) => s.stopped), composePath };
+}
+
+/** The stack's containers as `containers get` reports them, found by the stack's images and kept when Compose says they belong to it. */
+async function stackContainers(cli: Cli, stack: string, refs: readonly string[]): Promise<StackContainer[]> {
+    const listed = listRows(await cliJson(cli, ["containers", "list", "--all"])).filter((c) => typeof c.image === "string" && refs.includes(c.image));
+    const out: StackContainer[] = [];
+    for (const c of listed) {
+        if (typeof c.id !== "string") continue;
+        const name = Array.isArray(c.names) && typeof c.names[0] === "string" ? c.names[0].replace(/^\//, "") : c.id.slice(0, 12);
+        const facts = containerFacts(await cliJson(cli, ["containers", "get", c.id]), name);
+        if (facts.project === stack) out.push({ name, running: facts.running, state: facts.state, exitCode: facts.exitCode, health: facts.health });
+    }
+    return out;
+}
+
+/** Waits until the stack's sync reports success at `commit`; false when it does not within the limit. */
+async function waitForSync(config: ArcaneConfig, stack: string, commit: string): Promise<{ ok: boolean; last: string }> {
+    const deadline = Date.now() + STACK_SYNC_WAIT_MS;
+    for (;;) {
+        const sync = await withCli(config, async (cli) => projectSync(await cliJson(cli, ["projects", "list", "--all"]), await cliJson(cli, ["gitops", "list", "--all"]), stack));
+        const sameCommit = sync.commit !== null && (commit.startsWith(sync.commit) || sync.commit.startsWith(commit));
+        if (sameCommit && sync.status === "success") return { ok: true, last: `${sync.status} at ${sync.commit}` };
+        if (Date.now() > deadline) return { ok: false, last: `${sync.status ?? "none"} at ${sync.commit ?? "none"}` };
+        await new Promise((r) => setTimeout(r, 30_000));
+    }
+}
+
+async function redeploy(config: ArcaneConfig, stack: string): Promise<string | null> {
+    const result = await withCli(config, (cli) => cli(["projects", "redeploy", stack, "--yes"], REDEPLOY_TIMEOUT_MS));
+    return result.code === 0 ? null : (result.stderr.trim().split("\n").pop() ?? "").slice(0, 300) || `projects redeploy exited ${result.code}`;
+}
+
+/** The containers' problem after a redeploy, polled until none remains or the health wait ends. */
+async function settle(config: ArcaneConfig, stack: string, services: number): Promise<string | null> {
+    const deadline = Date.now() + HEALTH_WAIT_MS;
+    for (;;) {
+        const problem = await withCli(config, async (cli) => {
+            const refs = stackServices(await cliJson(cli, ["projects", "list", "--all"]))
+                .filter((s) => s.stack === stack)
+                .map((s) => s.ref)
+                .filter((r): r is string => r !== null);
+            return containersProblem(await stackContainers(cli, stack, refs), services);
+        });
+        if (problem === null || Date.now() > deadline) return problem;
+        await new Promise((r) => setTimeout(r, 15_000));
+    }
+}
+
+/**
+ * One stack's pending updates, end to end and without judgment: pinned
+ * references move in the Compose file through the GitOps repository, then the
+ * stack is redeployed with a pull and checked. Prints one JSON answer whose
+ * `status` is updated, rolled back, failed, or nothing to do; the exit code is
+ * 0 for all of them so the caller always gets the answer.
+ */
+async function stackUpdate(config: ArcaneConfig, args: string[]): Promise<number> {
+    const confirmed = args.includes("--confirm");
+    const rest = args.filter((a) => a !== "--confirm");
+    if (rest.length !== 1 || !NAME.test(rest[0])) throw new UsageError("stack-update <stack> [--confirm]");
+    const stack = rest[0];
+    if (stack === "arcane") throw new UsageError("arcane is the manager itself: use manager-upgrade (the Upgrade Arcane button)");
+    const spec = repoSpec(config);
+    const dir = await ensureCheckout(spec);
+    await git(spec, dir, ["pull", "--ff-only", "origin", config.gitopsBranch]);
+    const repo = await repoStatus(spec, dir);
+    if (repo.changed.length > 0) throw new ServiceError(`the arcane checkout has uncommitted changes: ${repo.changed.join(", ")}`);
+    if (repo.ahead > 0) throw new ServiceError(`the arcane checkout is ${repo.ahead} commit(s) ahead of origin`);
+    const holdsPath = join(dir, HOLDS_FILE);
+    const holds = existsSync(holdsPath) ? parseHolds(readFileSync(holdsPath, "utf-8")) : {};
+    if (holds[stack]) throw new UsageError(`${stack} is held in ${HOLDS_FILE}: ${holds[stack]}`);
+
+    const before = await withCli(config, (cli) => stackState(cli, stack, Date.now()));
+    if (before.stopped) throw new UsageError(`${stack} is stopped; a redeploy would start it, so start it on purpose first`);
+    const pending = pendingRows(before.rows);
+    const answer = (status: StackStatus, extra: Record<string, unknown>) => printJson({ status, stack, ...extra });
+    if (pending.length === 0) {
+        answer("nothing to do", { unknown: before.rows.filter((r) => r.status === "unknown").map((r) => ({ service: r.service, note: r.note })) });
+        return 0;
+    }
+    const majors = pending.filter((r) => crossesMajor(r.running, r.latest));
+    if (majors.length > 0 && !confirmed) {
+        throw new UsageError(`${majors.map((r) => `${r.service} ${r.running} -> ${r.latest}`).join(", ")} crosses a major version: summarise release-notes ${stack}/<service> for the user, then repeat with --confirm after their go`);
+    }
+    const results: ServiceResult[] = pending.map((r) => ({ service: r.service, ref: r.ref, from: r.running, to: r.latest, applied: nextImageRef(r) ?? "pull", result: "unchecked" }));
+    const edits = results.filter((r) => r.applied !== "pull");
+
+    let commit: string | null = null;
+    if (edits.length > 0) {
+        if (before.composePath === null) throw new ServiceError(`${stack} has no single GitOps sync with a Compose path`);
+        const file = safeRepoPath(dir, before.composePath);
+        let text = readFileSync(file, "utf-8");
+        const moves = new Map(edits.map((e) => [e.ref, e.applied]));
+        for (const [from, to] of moves) text = replaceImageRef(text, from, to);
+        writeFileSync(file, text);
+        await git(spec, dir, ["add", before.composePath]);
+        await git(spec, dir, ["commit", "--quiet", "-m", commitMessage(stack, edits.map((e) => ({ service: e.service, from: e.ref, to: e.applied })))]);
+        commit = (await git(spec, dir, ["rev-parse", "HEAD"])).trim();
+        await git(spec, dir, ["push", "origin", config.gitopsBranch]);
+        const synced = await waitForSync(config, stack, commit);
+        if (!synced.ok) {
+            await git(spec, dir, ["revert", "--no-edit", commit]);
+            await git(spec, dir, ["push", "origin", config.gitopsBranch]);
+            answer("failed", { reason: `the sync did not report success at ${commit.slice(0, 12)} within ${STACK_SYNC_WAIT_MS / MINUTE} minutes (last: ${synced.last}); the commit was reverted and nothing was redeployed`, commit, services: results });
+            return 0;
+        }
+    }
+
+    let problem = await redeploy(config, stack);
+    if (problem === null) problem = await settle(config, stack, before.services);
+    if (problem === null) {
+        const after = await withCli(config, (cli) => stackState(cli, stack, Date.now()));
+        for (const r of results) {
+            const now = after.rows.find((x) => x.service === r.service);
+            // Current when nothing newer is pending, or when the target runs and a release appeared meanwhile.
+            r.result = now !== undefined && (now.status === "up-to-date" || (now.status !== "unknown" && now.running === r.to)) ? "current" : "behind";
+        }
+        const behind = results.filter((r) => r.result === "behind").map((r) => r.service);
+        if (behind.length > 0) problem = `still behind after the redeploy: ${behind.join(", ")}`;
+    }
+    if (problem === null) {
+        answer("updated", { services: results, ...(commit ? { commit } : {}) });
+        return 0;
+    }
+    if (commit === null) {
+        answer("failed", { reason: problem, services: results, note: "the stack was redeployed with pulled images; a pull cannot be rolled back, so check the stack with the arcane specialist" });
+        return 0;
+    }
+    await git(spec, dir, ["revert", "--no-edit", commit]);
+    await git(spec, dir, ["push", "origin", config.gitopsBranch]);
+    const revert = (await git(spec, dir, ["rev-parse", "HEAD"])).trim();
+    const synced = await waitForSync(config, stack, revert);
+    const again = synced.ok ? await redeploy(config, stack) : `the revert's sync did not report success (last: ${synced.last})`;
+    const settled = again === null ? await settle(config, stack, before.services) : again;
+    if (settled === null) {
+        answer("rolled back", { reason: problem, services: results, commit, revert, note: "the previous Compose file runs again; images pulled for floating tags stay pulled" });
+        return 0;
+    }
+    answer("failed", { reason: `${problem}; the rollback did not finish either: ${settled}`, services: results, commit, revert });
+    return 0;
+}
+
 export async function command(args: string[]): Promise<number> {
     if (args.length === 0 || args[0] === "--help" || args[0] === "help") {
         printRaw(HELP);
         return 0;
     }
     const config = readConfig<ArcaneConfig>(SERVICE, SHAPE);
-    if (args[0] === "git") return repoCommand(repoSpec(config), args.slice(1));
+    if (args[0] === "git") return checkoutLocked(() => repoCommand(repoSpec(config), args.slice(1)));
     if (args[0] === "update-report") return updateReport(config, args.slice(1));
     if (args[0] === "release-notes") return releaseNotes(config, args.slice(1));
     if (args[0] === "host") return hostCommand(config, args.slice(1));
     if (args[0] === "project-status") return projectStatus(config, args.slice(1));
     if (args[0] === "deployment") return deployment(config, args.slice(1));
-    if (args[0] === "manager-upgrade") return managerUpgrade(config, args.slice(1));
+    if (args[0] === "manager-upgrade") return checkoutLocked(() => managerUpgrade(config, args.slice(1)));
     if (args[0] === "manager-rollback") return managerRollback(args.slice(1));
+    if (args[0] === "stack-update") return checkoutLocked(() => stackUpdate(config, args.slice(1)));
     return passthrough(config, args);
 }
 

package/specialists/edge/run.ts

@@ -15,7 +15,7 @@ const SERVICE = "edge";
 
 interface EdgeConfig extends Record<string, string | number> {
     gcloud: string;
-    /** Python 3.10+ interpreter for gcloud; the Homebrew cask installs none of its own. */
+    /** Python 3.10+ interpreter for gcloud: the installer's root-owned Python, never one the main user can replace. */
     gcloudPython: string;
     project: string;
     zone: string;
@@ -78,13 +78,13 @@ const PACKAGE = /^[a-z0-9][a-z0-9+.-]{0,63}$/;
  * public DNS the way users reach them: status code only, no redirect
  * followed, no credentials. `expect` holds the codes a healthy route returns
  * (the Supabase settings endpoint wants an API key, so 401; Studio redirects
- * to the Pomerium sign-in; the rest answer 200, as arcane-ops judged them).
+ * to the Pomerium sign-in; the rest answer 200).
  */
 export const PUBLIC_ROUTES: Readonly<Record<string, { url: string; expect: readonly number[] }>> = {
     "supabase-auth-settings": { url: "https://supabase.37pla.net/auth/v1/settings", expect: [401] },
     "supabase-jwks": { url: "https://supabase.37pla.net/auth/v1/.well-known/jwks.json", expect: [200] },
     "supabase-openid": { url: "https://supabase.37pla.net/auth/v1/.well-known/openid-configuration", expect: [200] },
-    "supabase-oauth-discovery": { url: "https://supabase.37pla.net/auth/v1/.well-known/oauth-authorization-server", expect: [200] },
+    "supabase-oauth-discovery": { url: "https://supabase.37pla.net/auth/v1/.well-known/oauth-authorization-server", expect: [404] },
     "supabase-studio": { url: "https://supabase-studio.37pla.net/", expect: [301, 302, 303, 307, 308] },
     "grimoire-public": { url: "https://cli.grimoire.monadeo.com/v1/auth/config", expect: [200] },
     "grimoire-public-health": { url: "https://cli.grimoire.monadeo.com/healthz", expect: [200] },

package/specialists/entry/remote/spc-arcane-install.sh

@@ -1,8 +1,8 @@
 #!/bin/sh
 # One-time setup of the spc-arcane user on VM 100 for the arcane specialist.
 # Run as root with these files in /tmp/spc-arcane: spc-arcane-host-entry,
-# spc-arcane-op, spc-arcane-sudoers, spc-arcane-host.pub, and the arcane-ops
-# probes host_probe.py, resource_probe.py, incident_probe.py, kernel_probe.py,
+# spc-arcane-op, spc-arcane-sudoers, spc-arcane-host.pub, and the
+# probes from host/spc-arcane-probes/: host_probe.py, resource_probe.py, incident_probe.py, kernel_probe.py,
 # storage_probe.py, queue_probe.py, mealie_probe.py, radarr_probe.py.
 # Idempotent; reinstalling updates the files and keeps the user.
 set -eu

package/specialists/entry/remote/spc-arcane-op.py

@@ -68,6 +68,9 @@ HOST_FILES = (
     ("spc-arcane-host-entry", "/usr/local/sbin/spc-arcane-host-entry"),
     ("spc-arcane-op", "/usr/local/sbin/spc-arcane-op"),
     ("spc-arcane-sudoers", "/etc/sudoers.d/spc-arcane"),
+) + tuple(
+    (f"spc-arcane-probes/{name}.py", f"{PROBES}/{name}.py")
+    for name in ("host_probe", "resource_probe", "incident_probe", "kernel_probe", "storage_probe", "queue_probe", "mealie_probe", "radarr_probe")
 )
 UNITS = tuple(name for name, _ in HOST_FILES if name.endswith((".service", ".timer")))
 LOOPBACK_CHECKS = {