@zibby/skills 0.1.95 → 0.2.0

package/docs/self-host/storage.md

@@ -0,0 +1,50 @@
+---
+sidebar_position: 4
+title: Storage
+---
+
+# Storage
+
+A self-hosted box keeps two stateful stores, both as single-box containers on named docker volumes:
+
+- **ScyllaDB Alternator** (`zibby-scylla`, volume `scylla-data`) — the DynamoDB-compatible metadata store. All accounts, projects, agents, integrations, executions, and stores live here.
+- **Object store** (`zibby-minio`, volume `seaweed-data` or legacy `minio-data`) — run sources, artifacts, per-node logs, datasets, and marketplace icons, over the S3 API.
+
+Both survive `docker compose restart` / `down` (they're on named volumes) but **not** `down -v`. Back them up — see [Backup & restore](./backup-restore).
+
+## MinIO → SeaweedFS
+
+MinIO's upstream was archived in 2026-04, so as of **0.1.126** the object store is **SeaweedFS**. The service is still *named* `minio` in compose and DNS so every `S3_ENDPOINT` consumer is unchanged — only the engine behind it changed. (Full evaluation: `strategy/object-store-replacement.md` in the root repo.)
+
+- **New installs** come up on SeaweedFS (volume `seaweed-data`). Nothing to do.
+- **Upgrades from a MinIO box auto-migrate.** When the installer upgrades a legacy (pre-flip, `minio-data`) box, it keeps MinIO running for that `up`, then — once the upgraded stack is healthy — runs the migration automatically.
+
+### How the auto-migration works
+
+It is designed to **auto-attempt, never auto-brick** — MinIO's data is never modified, so every failure path rolls back to a working box:
+
+1. **backup** — a full `backup.sh` snapshot first; it refuses to migrate without one.
+2. **quiesce** — stop the S3 writers (control-plane + copilot-runtime) so the copy is consistent.
+3. **copy + verify** — `migrate-object-store.sh` mirrors every bucket into a throwaway SeaweedFS container writing to the `seaweed-data` volume, then verifies per-bucket **object count + total bytes must match exactly**, plus a sha256 comparison of sampled objects (always including the largest). It never deletes anything.
+4. **flip** — record `ZIBBY_OBJECT_STORE=seaweedfs` in `.env` and bring up the default (SeaweedFS) stack.
+5. **health-gate** — the new control plane must become healthy **and** pass an authenticated S3 list against the migrated store.
+6. **rollback** — any failure at any stage rolls back to MinIO (whose data was never touched) with a loud status line.
+
+The **old `minio-data` volume is retained** as a rollback safety net. After a few days of verified operation you can reclaim it:
+
+```bash
+docker volume rm <project>_minio-data   # irreversible — only after you trust SeaweedFS
+```
+
+### Controlling / retrying the migration
+
+- **Opt out** (stay on MinIO for now): run the installer with `ZIBBY_SKIP_OBJECT_STORE_MIGRATION=1`.
+- **Retry** after a failed auto-migration: re-run the installer, or manually —
+  ```bash
+  cd zibby-selfhosted
+  docker compose stop control-plane copilot-runtime
+  bash migrate-object-store.sh --yes
+  # then flip: set ZIBBY_OBJECT_STORE=seaweedfs + COMPOSE_FILE=docker-compose.yml in .env
+  docker compose -f docker-compose.yml up -d
+  ```
+- **MinIO emergency fallback image** (if a legacy box lost its local MinIO image): the pinned tarball is at `https://dl.zibby.app/selfhosted/infra/` — verify the `.sha256`, then `docker load -i`. The box only ever pulls images from our own CDN, never a third-party registry.

package/docs/self-host/troubleshooting.md

@@ -0,0 +1,48 @@
+---
+sidebar_position: 5
+title: Troubleshooting
+---
+
+# Troubleshooting
+
+A self-hosted box is **one VM** — control-plane container + Scylla + object store + ephemeral run containers. When something is slow or failing, diagnose from the box's own logs; the reflex "it doesn't scale / the VM is overloaded" is almost always wrong (a `top` usually shows the box near-idle).
+
+## First moves
+
+```bash
+top                                          # CPU / memory — usually reveals it's idle
+docker ps -a                                 # are containers Up? did a run container Exit?
+docker logs zibby-control-plane --tail 100   # the executor + reaper + API
+docker logs zibby-bootstrap                  # first-boot template sync + the seeded token
+curl -s http://localhost:3001/health         # control-plane health
+curl -s http://localhost:3001/selfhost/version   # the running release (baked, authoritative)
+```
+
+To query the box's own API, use its `PROJECT_API_TOKEN` from `.env`:
+
+```bash
+curl -H "Authorization: Bearer <zby_...>" \
+  "http://localhost:3001/executions?projectId=selfhosted-project"
+```
+
+A failed run's `reapedReason` field tells you why it was reaped. (The cloud Zibby MCP/CLI talks to the *cloud* backend — it cannot reach a self-host box; you need the box's own token.)
+
+## Failure signatures worth recognizing
+
+**Runs "fail" with empty steps + `pull access denied for zibby-agent … repository does not exist`.** An agent-image tag mismatch after an upgrade — the executor is launching a tag the reused `.env` doesn't have. Fixed at the root in installer 0.1.53+; immediate unblock:
+
+```bash
+docker tag zibby-agent:slim zibby-agent:local
+```
+
+**The dashboard shows the new version but the API still behaves like the old one.** `docker compose up -d` can judge a service "up to date" when only the image *content* changed. The installer force-recreates the control-plane container to avoid this; if you hit it, re-run the installer (or `docker compose up -d --no-deps --force-recreate control-plane`). Because `/selfhost/version` reads the **baked** file, a genuine version mismatch is *visible* rather than hidden behind a lying env-var label.
+
+**Control plane won't become healthy right after boot.** Scylla cold-boots slowly (~1–2 min). The installer waits up to ~5 min; if it's still unhealthy, read `docker logs zibby-control-plane` — a broken data path shows up there, not in `top`.
+
+**An upgrade ended at `rolled_back`.** The box is safe (back on the prior version). Read `GET /selfhost/upgrade/status` `logTail` for the failing phase, and check `docker logs zibby-control-plane`. Re-attempt with a **pinned** version (see [Upgrade & rollback](./upgrade)).
+
+## Where things live
+
+- **Data** — docker volumes `scylla-data` + the object store volume. Back them up (see [Backup & restore](./backup-restore)); they survive restarts but not `down -v`.
+- **`.env`** — the runtime source of truth (`JWT_SECRET`, `ENCRYPTION_KEY`, tokens, `MAX_CONCURRENT_RUNS`, `AGENT_IMAGE`). Reused untouched on upgrade.
+- **Upgrade backups + logs** — `<install-dir>/backups/upgrade-<id>/`.

package/docs/self-host/upgrade.md

@@ -0,0 +1,81 @@
+---
+sidebar_position: 2
+title: Upgrade & rollback
+---
+
+# Upgrade & rollback
+
+## Check the running version
+
+```bash
+curl -s http://<host>:3001/selfhost/version
+# {"version":"0.1.126"}
+```
+
+This reads the **immutable file `/app/SELFHOST_VERSION`**, baked into the image at build time from `selfhosted/package.json`. It ships inside the image layer, so it **always matches the code actually running** and cannot be spoofed by a runtime env / compose `.env` override. That is the point: if an upgrade did not actually swap the control-plane image (a silent-upgrade failure), the baked file still reports the old version — you *see* the mismatch instead of trusting a lying label.
+
+The dashboard renders this same value as a small `v0.1.126` label in the sidebar, so the running release is always visible. (The dashboard has no upgrade button — upgrades are driven by the API endpoint or by re-running the installer, below.)
+
+The latest published version is in the release manifest:
+
+```bash
+curl -s https://dl.zibby.app/selfhosted/latest/manifest.env
+# ZIBBY_SELFHOSTED_VERSION=0.1.126
+# IMAGE_TARBALL=zibby-selfhosted-0.1.126.tar.gz
+# ...
+```
+
+## Two ways to upgrade
+
+### 1. Re-run the installer (in place)
+
+The installer is idempotent — re-running it pulls a newer release, verifies + `docker load`s the new images, re-pins `AGENT_IMAGE`, and **force-recreates** the control-plane container from the freshly loaded image:
+
+```bash
+curl -fsSL https://dl.zibby.app/selfhosted/latest/install.sh | bash
+# or, from the install dir:  cd zibby-selfhosted && bash install.sh
+```
+
+Your `.env` (JWT_SECRET, ENCRYPTION_KEY, tokens) and data volumes are preserved.
+
+### 2. The owner-only remote upgrade (recommended)
+
+`POST /selfhost/upgrade` runs a **safe, self-driving upgrade** with an automatic backup and rollback — you can trigger it remotely and it survives the control-plane restart. It is **owner-only**: the endpoint requires the workspace-owner PAT/session or the admin project token (a run token or a non-owner member is refused `403`).
+
+```bash
+curl -s -X POST http://<host>:3001/selfhost/upgrade \
+  -H "Authorization: Bearer <owner-or-admin-token>" \
+  -H "Content-Type: application/json" \
+  -d '{"version":"0.1.126"}'
+# → 202 { "upgradeId": "...", "phase": "starting", ... }
+```
+
+Because a container can't recreate itself, the control plane launches a **detached helper** container (with the host docker socket) that outlives the control-plane bounce and drives the upgrade end to end:
+
+1. **backup** — flush Scylla, tar the Scylla + object-store volumes, copy `.env`, into `<install-dir>/backups/upgrade-<id>/`.
+2. **anchor** — pin the current control-plane image as `zibby-control-plane:rollback-<id>` so the new `docker load` can't garbage-collect the rollback target.
+3. **download → install** — fetch the target release's `install.sh` (Ed25519 + checksum verified), `docker load`, re-pin `AGENT_IMAGE`, force-recreate.
+4. **healthcheck** — poll the new control plane until container-healthy **and** `GET /selfhost/version` returns the target version.
+5. **rollback** — on an unhealthy new control plane (or an installer that exited non-zero), re-tag the anchor back to `:local` and force-recreate → back to the working version.
+
+Poll the status (owner-only, read-only):
+
+```bash
+curl -s http://<host>:3001/selfhost/upgrade/status \
+  -H "Authorization: Bearer <owner-or-admin-token>"
+# { "upgrade": { "phase": "install", ... }, "logTail": "...", "history": [...] }
+```
+
+Phases progress `starting → backup → anchor → download → install → healthcheck → success`. A failed upgrade ends at `rolled_back` (the box is back on the prior, healthy version) or, if even rollback fails, `rollback_failed` (manual restore needed — the backup dir and rollback-anchor tag are named in the status). The control plane restarts mid-upgrade, so the poll may briefly fail to connect — that is expected.
+
+:::warning Pin the exact version, not `latest`
+`{"version":"latest"}` resolves through the CDN, which can serve a **stale cached** `latest` for a while after a publish. For a deterministic upgrade, pass the exact version you verified against `manifest.env` (e.g. `{"version":"0.1.126"}`). The helper also treats "healthy but the version didn't advance to the pinned target" as a failure and rolls back — a guard you only get with a pinned version.
+:::
+
+## If a run fails with empty steps after an upgrade
+
+Symptom: reviews/runs "fail" with `steps: []` and the control-plane log shows `pull access denied for zibby-agent … repository does not exist`. That is an **agent-image tag mismatch** — the executor is trying to launch a tag the reused `.env` points at but the tarball loaded a different one. Installer 0.1.53+ fixes this at the root (it pins `AGENT_IMAGE` to the loaded tag and aliases `zibby-agent:local`). Immediate operator unblock:
+
+```bash
+docker tag zibby-agent:slim zibby-agent:local
+```

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@zibby/skills",
-  "version": "0.1.95",
+  "version": "0.2.0",
   "description": "Built-in skill definitions for the Zibby agent-workflow framework",
   "type": "module",
   "main": "dist/index.js",
@@ -141,6 +141,7 @@
     "esbuild": "^0.28.0",
     "eslint": "^10.0.2",
     "globals": "^17.4.0",
-    "typescript": "^5.9.3"
+    "typescript": "^5.9.3",
+    "@types/node": "^20.14.0"
   }
 }