@zibby/skills 0.1.96 → 0.2.0

package/docs/self-host/troubleshooting.md

@@ -0,0 +1,48 @@
+---
+sidebar_position: 5
+title: Troubleshooting
+---
+
+# Troubleshooting
+
+A self-hosted box is **one VM** — control-plane container + Scylla + object store + ephemeral run containers. When something is slow or failing, diagnose from the box's own logs; the reflex "it doesn't scale / the VM is overloaded" is almost always wrong (a `top` usually shows the box near-idle).
+
+## First moves
+
+```bash
+top                                          # CPU / memory — usually reveals it's idle
+docker ps -a                                 # are containers Up? did a run container Exit?
+docker logs zibby-control-plane --tail 100   # the executor + reaper + API
+docker logs zibby-bootstrap                  # first-boot template sync + the seeded token
+curl -s http://localhost:3001/health         # control-plane health
+curl -s http://localhost:3001/selfhost/version   # the running release (baked, authoritative)
+```
+
+To query the box's own API, use its `PROJECT_API_TOKEN` from `.env`:
+
+```bash
+curl -H "Authorization: Bearer <zby_...>" \
+  "http://localhost:3001/executions?projectId=selfhosted-project"
+```
+
+A failed run's `reapedReason` field tells you why it was reaped. (The cloud Zibby MCP/CLI talks to the *cloud* backend — it cannot reach a self-host box; you need the box's own token.)
+
+## Failure signatures worth recognizing
+
+**Runs "fail" with empty steps + `pull access denied for zibby-agent … repository does not exist`.** An agent-image tag mismatch after an upgrade — the executor is launching a tag the reused `.env` doesn't have. Fixed at the root in installer 0.1.53+; immediate unblock:
+
+```bash
+docker tag zibby-agent:slim zibby-agent:local
+```
+
+**The dashboard shows the new version but the API still behaves like the old one.** `docker compose up -d` can judge a service "up to date" when only the image *content* changed. The installer force-recreates the control-plane container to avoid this; if you hit it, re-run the installer (or `docker compose up -d --no-deps --force-recreate control-plane`). Because `/selfhost/version` reads the **baked** file, a genuine version mismatch is *visible* rather than hidden behind a lying env-var label.
+
+**Control plane won't become healthy right after boot.** Scylla cold-boots slowly (~1–2 min). The installer waits up to ~5 min; if it's still unhealthy, read `docker logs zibby-control-plane` — a broken data path shows up there, not in `top`.
+
+**An upgrade ended at `rolled_back`.** The box is safe (back on the prior version). Read `GET /selfhost/upgrade/status` `logTail` for the failing phase, and check `docker logs zibby-control-plane`. Re-attempt with a **pinned** version (see [Upgrade & rollback](./upgrade)).
+
+## Where things live
+
+- **Data** — docker volumes `scylla-data` + the object store volume. Back them up (see [Backup & restore](./backup-restore)); they survive restarts but not `down -v`.
+- **`.env`** — the runtime source of truth (`JWT_SECRET`, `ENCRYPTION_KEY`, tokens, `MAX_CONCURRENT_RUNS`, `AGENT_IMAGE`). Reused untouched on upgrade.
+- **Upgrade backups + logs** — `<install-dir>/backups/upgrade-<id>/`.

package/docs/self-host/upgrade.md

@@ -0,0 +1,81 @@
+---
+sidebar_position: 2
+title: Upgrade & rollback
+---
+
+# Upgrade & rollback
+
+## Check the running version
+
+```bash
+curl -s http://<host>:3001/selfhost/version
+# {"version":"0.1.126"}
+```
+
+This reads the **immutable file `/app/SELFHOST_VERSION`**, baked into the image at build time from `selfhosted/package.json`. It ships inside the image layer, so it **always matches the code actually running** and cannot be spoofed by a runtime env / compose `.env` override. That is the point: if an upgrade did not actually swap the control-plane image (a silent-upgrade failure), the baked file still reports the old version — you *see* the mismatch instead of trusting a lying label.
+
+The dashboard renders this same value as a small `v0.1.126` label in the sidebar, so the running release is always visible. (The dashboard has no upgrade button — upgrades are driven by the API endpoint or by re-running the installer, below.)
+
+The latest published version is in the release manifest:
+
+```bash
+curl -s https://dl.zibby.app/selfhosted/latest/manifest.env
+# ZIBBY_SELFHOSTED_VERSION=0.1.126
+# IMAGE_TARBALL=zibby-selfhosted-0.1.126.tar.gz
+# ...
+```
+
+## Two ways to upgrade
+
+### 1. Re-run the installer (in place)
+
+The installer is idempotent — re-running it pulls a newer release, verifies + `docker load`s the new images, re-pins `AGENT_IMAGE`, and **force-recreates** the control-plane container from the freshly loaded image:
+
+```bash
+curl -fsSL https://dl.zibby.app/selfhosted/latest/install.sh | bash
+# or, from the install dir:  cd zibby-selfhosted && bash install.sh
+```
+
+Your `.env` (JWT_SECRET, ENCRYPTION_KEY, tokens) and data volumes are preserved.
+
+### 2. The owner-only remote upgrade (recommended)
+
+`POST /selfhost/upgrade` runs a **safe, self-driving upgrade** with an automatic backup and rollback — you can trigger it remotely and it survives the control-plane restart. It is **owner-only**: the endpoint requires the workspace-owner PAT/session or the admin project token (a run token or a non-owner member is refused `403`).
+
+```bash
+curl -s -X POST http://<host>:3001/selfhost/upgrade \
+  -H "Authorization: Bearer <owner-or-admin-token>" \
+  -H "Content-Type: application/json" \
+  -d '{"version":"0.1.126"}'
+# → 202 { "upgradeId": "...", "phase": "starting", ... }
+```
+
+Because a container can't recreate itself, the control plane launches a **detached helper** container (with the host docker socket) that outlives the control-plane bounce and drives the upgrade end to end:
+
+1. **backup** — flush Scylla, tar the Scylla + object-store volumes, copy `.env`, into `<install-dir>/backups/upgrade-<id>/`.
+2. **anchor** — pin the current control-plane image as `zibby-control-plane:rollback-<id>` so the new `docker load` can't garbage-collect the rollback target.
+3. **download → install** — fetch the target release's `install.sh` (Ed25519 + checksum verified), `docker load`, re-pin `AGENT_IMAGE`, force-recreate.
+4. **healthcheck** — poll the new control plane until container-healthy **and** `GET /selfhost/version` returns the target version.
+5. **rollback** — on an unhealthy new control plane (or an installer that exited non-zero), re-tag the anchor back to `:local` and force-recreate → back to the working version.
+
+Poll the status (owner-only, read-only):
+
+```bash
+curl -s http://<host>:3001/selfhost/upgrade/status \
+  -H "Authorization: Bearer <owner-or-admin-token>"
+# { "upgrade": { "phase": "install", ... }, "logTail": "...", "history": [...] }
+```
+
+Phases progress `starting → backup → anchor → download → install → healthcheck → success`. A failed upgrade ends at `rolled_back` (the box is back on the prior, healthy version) or, if even rollback fails, `rollback_failed` (manual restore needed — the backup dir and rollback-anchor tag are named in the status). The control plane restarts mid-upgrade, so the poll may briefly fail to connect — that is expected.
+
+:::warning Pin the exact version, not `latest`
+`{"version":"latest"}` resolves through the CDN, which can serve a **stale cached** `latest` for a while after a publish. For a deterministic upgrade, pass the exact version you verified against `manifest.env` (e.g. `{"version":"0.1.126"}`). The helper also treats "healthy but the version didn't advance to the pinned target" as a failure and rolls back — a guard you only get with a pinned version.
+:::
+
+## If a run fails with empty steps after an upgrade
+
+Symptom: reviews/runs "fail" with `steps: []` and the control-plane log shows `pull access denied for zibby-agent … repository does not exist`. That is an **agent-image tag mismatch** — the executor is trying to launch a tag the reused `.env` points at but the tarball loaded a different one. Installer 0.1.53+ fixes this at the root (it pins `AGENT_IMAGE` to the loaded tag and aliases `zibby-agent:local`). Immediate operator unblock:
+
+```bash
+docker tag zibby-agent:slim zibby-agent:local
+```

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@zibby/skills",
-  "version": "0.1.96",
+  "version": "0.2.0",
   "description": "Built-in skill definitions for the Zibby agent-workflow framework",
   "type": "module",
   "main": "dist/index.js",
@@ -141,6 +141,7 @@
     "esbuild": "^0.28.0",
     "eslint": "^10.0.2",
     "globals": "^17.4.0",
-    "typescript": "^5.9.3"
+    "typescript": "^5.9.3",
+    "@types/node": "^20.14.0"
   }
 }