@bongos/core 1.17.0

package/docs/recipes/gds-db-backup.md

@@ -0,0 +1,255 @@
+# Recipe — GDS DB backup (local nightly + offsite weekly)
+
+> **Filed by:** task [#412](https://example.com/builders#/task/412) (V3.R62-precondition, local nightly) + [#415](https://example.com/builders#/task/415) (V3.R62-offsite, weekly offsite archive) — both under [ADR 0024](../adr/<redacted>.md) §5 (backup cadence). Vendor choice for the offsite copy: [ADR 0025](../adr/<redacted>.md) (DigitalOcean Spaces).
+> **Two layers:** a **local daily** `pg_dump` to `/var/backups/gds/` (protects against corruption / bad migration / accidental delete) **plus** a **weekly offsite copy** of the latest local dump to a DigitalOcean Space (protects against whole-droplet loss).
+> **Reads:** `/etc/systemd/system/gds-db-backup.{service,timer}` (copies of [`scripts/gds/gds-db-backup.service`](../../scripts/gds/gds-db-backup.service) / [`.timer`](../../scripts/gds/gds-db-backup.timer)) + [`scripts/gds/db-backup-nightly.sh`](../../scripts/gds/db-backup-nightly.sh); `/etc/systemd/system/gds-db-offsite.{service,timer}` (copies of [`scripts/gds/gds-db-offsite.service`](../../scripts/gds/gds-db-offsite.service) / [`.timer`](../../scripts/gds/gds-db-offsite.timer)) + [`scripts/gds/db-backup-offsite.sh`](../../scripts/gds/db-backup-offsite.sh).
+
+## What this gives us
+
+A nightly `pg_dump` of the `example` Postgres database, written to `/var/backups/gds/` on the droplet as gzipped plain-SQL dumps. Files older than 30 days are auto-pruned at the end of each successful run, so the directory holds approximately the last 30 days of dumps with no manual intervention.
+
+This closes the **most likely** loss-of-data risks: schema corruption, an accidental `DROP`/`DELETE`, a bad migration, a Postgres upgrade gone wrong, or someone (probably an LLM) writing a row they shouldn't have. It does **not** close whole-droplet loss — see the "Limits" section below and [task #415](https://status.example.com).
+
+## Where dumps live
+
+```
+/var/backups/gds/
+├── example-2026-05-29T040000Z.sql.gz   ← latest
+├── example-2026-05-28T040000Z.sql.gz
+└── …
+```
+
+- **Filename format:** `<dbname>-<UTC-timestamp>.sql.gz`. UTC is used so the dumps sort chronologically regardless of the droplet's timezone.
+- **Permissions:** dir is `lars:lars 0750`; files are `lars:lars 0640`. No root needed for routine inspection.
+- **Size:** each gzipped dump is well under 1 MB, and 30 days of them fit comfortably — pruning is hygiene, not disk pressure. **The dump excludes `search_chunks` data** (the `/recall` lexical index, migration 103) via `--exclude-table-data=search_chunks` ([#973](https://example.com/builders#/task/973)): it is regenerable derived data — chunked copies of DB prose (already dumped) and repo markdown (already in git) — and dumping its rows roughly duplicated the prose corpus, making it the dump's dominant growth vector. The table schema + GIN indexes are still dumped, so a restore recreates it empty; the index rebuilds from source (see the restore drill below). A size guard in the backup script WARNs to journald if a dump ever exceeds `GDS_BACKUP_WARN_BYTES` (default 25 MiB) — a generous tripwire for unexpected growth from some *other* table.
+
+## What's scheduled
+
+```
+gds-db-backup.timer ──→ gds-db-backup.service ──→ db-backup-nightly.sh
+   (daily 04:00 UTC)        (oneshot, as lars)         (pg_dump + gzip + prune)
+```
+
+- **Schedule:** every day at 04:00 UTC, with up to 5 minutes of jitter (`RandomizedDelaySec=300`). Low-traffic window for a US-East droplet.
+- **Catch-up:** if the droplet was offline when a run was due, `Persistent=true` runs it on next boot. So a maintenance window doesn't silently skip a day.
+- **Hardening:** the unit runs with `NoNewPrivileges`, `ProtectSystem=strict`, and `ReadWritePaths=/var/backups/gds`. The script can write dumps but otherwise can't modify the filesystem.
+
+## Offsite weekly archive ([#415](https://example.com/builders#/task/415), ADR 0025)
+
+The local dumps above live on the same disk as the database. The **offsite** layer copies the most recent local dump to a **DigitalOcean Space** (S3-compatible object storage) once a week, so a whole-droplet loss costs at most ~1 week of writes.
+
+```
+gds-db-offsite.timer ──→ gds-db-offsite.service ──→ db-backup-offsite.sh
+   (weekly Sun 05:00 UTC)    (oneshot, as lars)        (find latest local dump → upload to Space → prune)
+```
+
+- **It does not take its own `pg_dump`.** It uploads the newest `/var/backups/gds/<db>-*.sql.gz` that the local nightly already produced. Runs at 05:00 UTC — one hour after the 04:00 local dump — so the copy is same-day-fresh.
+- **Bucket layout:** objects land under the `gds-offsite/` key prefix in the Space.
+- **Retention:** 4 weekly copies (`GDS_OFFSITE_KEEP=4` — ~1 month), pruned oldest-first under that prefix. Sub-1 MB each.
+- **Credentials** live at `/etc/example/spaces.env` (chmod 600, owned by `lars`), never in the repo (ADR 0022). Required vars: `SPACES_KEY`, `SPACES_SECRET`, `SPACES_BUCKET`, `SPACES_REGION`, `SPACES_ENDPOINT`. systemd injects them via `EnvironmentFile`; a manual run sources the same file.
+- **Dependency:** `python3` + the `boto3` module on the droplet. The script fails loudly if either is missing.
+
+> **History:** an earlier task ([#7](https://example.com/builders#/task/7)) shipped a `scripts/backup.sh` + `/etc/cron.d/otb-backup` daily-to-Spaces cron that was forgotten and never worked (empty credentials, no Space, no log). [#415](https://example.com/builders#/task/415) retired it and replaced it with the systemd unit above. If you find any trace of `otb-backup.cron`, `scripts/backup.sh`, or `~/.config/otb/spaces.env` on a host, it's dead — remove it.
+>
+> **Retirement completed 2026-06-01 ([#529](https://example.com/builders#/task/529)):** [#415](https://example.com/builders#/task/415)'s retirement was mechanism-only — the leftover artifacts were *still present* and were purged in this pass: `/etc/cron.d/otb-backup` (a silent no-op — its target script `scripts/backup.sh` was never committed, so `deploy.sh`'s git-sync never placed it on the droplet, and `/var/log/otb-backup.log` was never created) and the dead `~/.config/otb/spaces.env` creds on the droplet, plus untracked copies of `scripts/backup.sh` + `infra/backup/otb-backup.cron` in the shared main checkout (the latter cleared under [#527](https://example.com/builders#/task/527) — they were also tripping `mainWorktreeBusy()` and bailing every auto-merge). **There is no offsite gap:** both layers documented above are live and verified — `gds-db-backup` (local daily) and `gds-db-offsite` (weekly, last run confirmed uploading). The dead prototype is the *only* thing that used the old `~/.config/otb/spaces.env` path — a *different* path from the live offsite creds documented above; do not recreate it.
+
+### First-time install of the offsite layer
+
+```bash
+# 1. Drop the Spaces credentials into place (values from the DO console:
+#    Spaces bucket name + region + a generated Spaces access key/secret).
+sudo install -d -o lars -g lars -m 0750 /etc/example
+sudo -u lars tee /etc/example/spaces.env >/dev/null <<'ENV'
+SPACES_KEY=...
+SPACES_SECRET=...
+SPACES_BUCKET=otb-gds-backups
+SPACES_REGION=nyc3
+SPACES_ENDPOINT=https://nyc3.digitaloceanspaces.com
+ENV
+sudo chmod 600 /etc/example/spaces.env
+
+# 2. Ensure boto3 is available.
+python3 -c "import boto3" 2>/dev/null || sudo apt-get install -y python3-boto3
+
+# 3. Install the systemd units (canonical text is in the repo).
+sudo install -o root -g root -m 0644 \
+  /home/lars/example/scripts/gds/gds-db-offsite.service \
+  /etc/systemd/system/gds-db-offsite.service
+sudo install -o root -g root -m 0644 \
+  /home/lars/example/scripts/gds/gds-db-offsite.timer \
+  /etc/systemd/system/gds-db-offsite.timer
+
+# 4. Reload, enable + start the timer, verify, and do a one-shot test upload.
+sudo systemctl daemon-reload
+sudo systemctl enable --now gds-db-offsite.timer
+systemctl list-timers gds-db-offsite.timer --no-pager
+sudo systemctl start gds-db-offsite.service && journalctl -u gds-db-offsite.service -n 20 --no-pager
+```
+
+## First-time install (already done; here for re-install / staging)
+
+```bash
+# 1. Create the destination directory.
+sudo install -d -o lars -g lars -m 0750 /var/backups/gds
+
+# 2. Drop the systemd units into place (canonical text is in the repo).
+sudo install -o root -g root -m 0644 \
+  /home/lars/example/scripts/gds/gds-db-backup.service \
+  /etc/systemd/system/gds-db-backup.service
+sudo install -o root -g root -m 0644 \
+  /home/lars/example/scripts/gds/gds-db-backup.timer \
+  /etc/systemd/system/gds-db-backup.timer
+
+# 3. Make sure the script is executable (it is in the repo, but a fresh
+# install on a different machine sometimes loses the bit).
+chmod +x /home/lars/example/scripts/gds/db-backup-nightly.sh
+
+# 4. Reload systemd, enable + start the timer.
+sudo systemctl daemon-reload
+sudo systemctl enable --now gds-db-backup.timer
+
+# 5. Verify the timer is armed.
+systemctl list-timers gds-db-backup.timer --no-pager
+```
+
+## Routine operations
+
+### Run a one-shot dump manually
+
+```bash
+sudo systemctl start gds-db-backup.service
+# Watch it complete:
+journalctl -u gds-db-backup.service -f
+```
+
+Or directly:
+
+```bash
+/home/lars/example/scripts/gds/db-backup-nightly.sh
+```
+
+### See when the next run is scheduled
+
+```bash
+systemctl list-timers gds-db-backup.timer --no-pager
+# NEXT                         LEFT     LAST                         PASSED  UNIT
+# Sat 2026-05-30 04:00:00 UTC  5h 12m   Fri 2026-05-29 04:00:00 UTC   19h     gds-db-backup.timer
+```
+
+### See history of past runs
+
+```bash
+journalctl -u gds-db-backup.service --since "1 week ago"
+```
+
+Each successful run emits one line:
+
+```
+gds-db-backup: ok · 28 dump(s) retained · 612345 bytes latest
+```
+
+A failed run exits non-zero, which systemd records — set up `journalctl -u gds-db-backup.service --no-pager | grep -i error` if you want to skim for trouble.
+
+### Inspect a dump without restoring
+
+```bash
+# Headers + table list, no actual restore.
+gunzip -c /var/backups/gds/example-<ts>.sql.gz | head -50
+
+# Or, for a custom-format dump (we don't generate these — but if a future
+# variant does), use pg_restore --list to enumerate objects:
+#   pg_restore --list /var/backups/gds/<file>
+```
+
+## Restore drills
+
+### Test-restore into a sandbox schema (preferred — non-destructive)
+
+```bash
+# Create an empty sandbox database for restore testing.
+sudo -u postgres createdb example_restore_test
+
+# Restore the most recent dump into it.
+gunzip -c /var/backups/gds/$(ls -1t /var/backups/gds/example-*.sql.gz | head -1) \
+  | sudo -u postgres psql -d example_restore_test
+
+# Spot-check: row counts on a few tables should be sane.
+sudo -u postgres psql -d example_restore_test -c \
+  "SELECT 'tasks' as tbl, count(*) FROM tasks
+   UNION ALL SELECT 'builders', count(*) FROM builders
+   UNION ALL SELECT 'claims', count(*) FROM claims;"
+
+# Cleanup.
+sudo -u postgres dropdb example_restore_test
+```
+
+> **Expected after restore: `search_chunks` is empty.** The dump excludes its data by design (see the Size note above), so `SELECT count(*) FROM search_chunks` returns 0 on a fresh restore — that is correct, not a truncated dump. `/recall` rebuilds the index from source once the app is back: `search-ingest-md.js` runs on the next deploy (repo-doc chunks) and `search-ingest-db.js` runs nightly (DB-prose chunks). For a real (not drill) restore where you want recall working immediately, run both by hand against the restored DB:
+> ```bash
+> node scripts/gds/search-ingest-md.js && node scripts/gds/search-ingest-db.js
+> ```
+
+A run-through of this drill is what shipped task [#281](https://example.com/builders#/task/281) (R62) once. Cadence going forward: quarterly.
+
+### Restore-from-Spaces drill (verifies the OFFSITE copy is usable)
+
+The point of the offsite copy is that it works when the droplet's disk is gone — so the drill must pull from the Space, not from `/var/backups/gds/`. Quarterly:
+
+```bash
+# Pull the latest offsite object down to /tmp using the same creds the
+# uploader uses. (boto3 one-liner so there's no extra CLI dependency.)
+set -a; source /etc/example/spaces.env; set +a
+python3 - <<'PY'
+import os, boto3
+s3 = boto3.client("s3", region_name=os.environ["SPACES_REGION"],
+    endpoint_url=os.environ["SPACES_ENDPOINT"],
+    aws_access_key_id=<redacted>"SPACES_KEY"],
+    aws_secret_access_key=<redacted>"SPACES_SECRET"])
+objs = sorted([o for o in s3.list_objects_v2(Bucket=os.environ["SPACES_BUCKET"],
+    Prefix="gds-offsite/").get("Contents", []) if o["Key"].endswith(".sql.gz")],
+    key=lambda o: o["LastModified"])
+key = objs[-1]["Key"]; dest = "/tmp/" + key.split("/")[-1]
+s3.download_file(os.environ["SPACES_BUCKET"], key, dest)
+print("downloaded", key, "→", dest)
+PY
+
+# Restore it into a sandbox DB and spot-check, then clean up.
+sudo -u postgres createdb example_offsite_test
+gunzip -c /tmp/example-*.sql.gz | sudo -u postgres psql -d example_offsite_test
+sudo -u postgres psql -d example_offsite_test -c "SELECT count(*) FROM tasks;"
+sudo -u postgres dropdb example_offsite_test && rm -f /tmp/example-*.sql.gz
+```
+
+### Full-restore of the live database (DESTRUCTIVE — last-resort recovery)
+
+> ⚠️ Stop the application before running this. The live DB will be wiped and replaced.
+
+```bash
+# 1. Stop the app so no writes are happening.
+sudo systemctl stop example
+
+# 2. Pick the dump to restore.
+DUMP=/var/backups/gds/example-<ts>.sql.gz
+
+# 3. Restore. The dump was created with --clean --if-exists, so it drops
+# existing objects before recreating them — no manual cleanup needed.
+gunzip -c "$DUMP" | sudo -u postgres psql -d example
+
+# 4. Restart the app.
+sudo systemctl start example
+
+# 5. Smoke: hit /api/gds/healthz, /api/gds/versions, /api/gds/public/progress.
+curl -s https://example.com/api/gds/healthz
+curl -s https://example.com/api/gds/public/progress | jq '.progress[0]'
+```
+
+## Limits — what this recipe does NOT cover
+
+1. **Whole-droplet loss window is ~1 week, not zero.** The weekly offsite copy ([#415](https://example.com/builders#/task/415), above) closes the catastrophic-loss gap, but it runs weekly — so in the worst case (droplet dies right before the weekly run) you lose up to ~7 days of writes that made it into the local daily dumps but not yet offsite. Acceptable for the GDS DB's shape; tighten the offsite cadence to daily if that ever stops being true.
+2. **Continuous archiving / PITR is not configured.** A restore brings you back to the most recent nightly snapshot, not the precise moment of failure. Up to 24 hours of writes can be lost in a worst-case restore. That's acceptable for the GDS DB's current shape (work-state ledger, no transactional money flows) but worth revisiting if/when the DB takes on stripe-payment or fulfillment state.
+3. **No automated alerting on a failed run yet.** The unit emits a non-zero exit on failure, which `systemctl status gds-db-backup.timer` will surface, but nothing pages you. Add a check to the existing `overnight-builder` health surface or the `weekly-security-review` scheduled task if backups silently failing for N days becomes a real concern.
+4. **No automated cadence for the restore drill.** The recipe above is just text until a human (or scheduled task) runs it. [#281](https://example.com/builders#/task/281) R62 is where the formal drill cadence lands.
+
+## Cost
+
+- **Local nightly ([#412](https://example.com/builders#/task/412)):** $0/mo. `/var/backups/gds/` lives on the existing droplet disk; dumps are sub-MB; the systemd timer is free. The droplet is 67 GB total; years of dumps would not be a sizing concern.
+- **Offsite weekly ([#415](https://example.com/builders#/task/415)):** $5/mo flat — a DigitalOcean Space (250 GB + 1 TB transfer included). We retain only 4 sub-MB objects, so we use a rounding error of the tier. The $5/mo over Backblaze B2's ~$0 is a deliberate one-vendor / same-region simplicity choice — see [ADR 0025](../adr/<redacted>.md).

package/docs/recipes/instance-deploy.md

@@ -0,0 +1,37 @@
+# Recipe — deploying a Cloud Bongos instance on a pinned core (`git reset` → `npm ci`)
+
+> ADR [0108](../adr/<redacted>.md) §4 (configurable-root W4, task [#1884](https://example.com/builders#/task/1884)). How a **consumer instance** — one that installs `@cloudbongos/core` as a versioned npm dependency rather than living in the monorepo — deploys. This is the target the live droplet deploy converts to; the reference implementation is [`scripts/deploy/deploy-instance.sh`](../../scripts/deploy/deploy-instance.sh).
+
+## The two models
+
+| | Pre-split (today, `example`) | Post-split (consumer instance) |
+|---|---|---|
+| Core code | in-tree (the monorepo) | `node_modules/@cloudbongos/core` (a pinned version) |
+| Deploy | [`deploy-prod.sh`](../../scripts/deploy/deploy-prod.sh): `git reset --hard origin/main` on a shared checkout | [`deploy-instance.sh`](../../scripts/deploy/deploy-instance.sh): `git pull` the instance repo → `npm ci` → migrate → restart |
+| Upgrade the core | (n/a — one tree) | bump the pin in `package.json` → `npm ci` → migrate → restart (ADR 0108 §4; the R86 channel, task [#1690](https://example.com/builders#/task/1690)) |
+| `cloudbongos.com` | a 2nd service on the **same** checkout (why it ran stale) | its **own** consumer repo + its own `deploy-instance.sh` |
+
+## The flow (`deploy-instance.sh`)
+
+1. **`git pull --ff-only origin main`** the *instance* repo — host content only: `config/`, `migrations/instance/`, and the pinned core version in `package.json`.
+2. **`npm ci --omit=dev`** — installs the pinned `@cloudbongos/core` + instance deps. This is what replaces `git reset --hard`: the core is a dependency, so an upgrade is a pin bump + `npm ci`, not a source merge.
+3. **`bash <core>/scripts/migrate.sh`** — the core's **three-root** migrate (W5 / ADR 0108 §5) merges `<core>/migrations` + `<instance>/migrations/instance` + each enabled module's migrations, keyed by namespaced filename stems (`core_NNN_`, `<key>_NNN_`, bare instance) so the two repos never collide. It resolves core-root vs instance-root via `src/instance-config.js`.
+4. **restart the platform** — a systemd unit runs `node <core>/src/platform-server.js` (GDS-only) with `WorkingDirectory` = the instance root (so `resolveInstanceRoot()` resolves to the instance) + `PORT`/`HOST` env; then poll `/healthz`.
+
+`docker-compose.yml` + `.github/workflows/selfhost-boot.yml` already model this shape (fresh Postgres → `npm ci` → `migrate.sh` → `platform-server`), so the flow is validated end-to-end there.
+
+## Status — ready, not yet the live path
+
+`deploy-instance.sh` is **additive and inactive**: the live `example` droplet still deploys via `deploy-prod.sh` (`git reset --hard`). The cutover is deliberately deferred:
+
+- **First run: Mercury (R85, task [#1689](https://example.com/builders#/task/1689))** — a greenfield consumer on a pinned core proves the flow.
+- **`example` cutover: R89 (task [#1728](https://example.com/builders#/task/1728))** — "Example splits last", **behind a security review** (ADR 0108 §6). It **must** land in lockstep with cutting the shared-checkout dual-instance deploy, or `cloudbongos.com` breaks (ADR 0108 §Consequences item 2).
+
+`ship.js` needs no change: its deploy **trigger** SSHes `~/deploy.sh` (or fires the CI workflow) and is agnostic to what that script does internally — a consumer instance installs `deploy-instance.sh` as its `~/deploy.sh`.
+
+## Wiring it on a consumer host (R85/R89)
+
+1. Install the script as the droplet's `~/deploy.sh` (the CI deploy key is forced-command-locked to `~/deploy.sh` — ADR 0043 — so it stays hand-installed, like `deploy-prod.sh`).
+2. Create a systemd unit for the instance: `ExecStart=/usr/bin/node <instanceRoot>/node_modules/@cloudbongos/core/src/platform-server.js`, `WorkingDirectory=<instanceRoot>`, `Environment=PORT=… HOST=127.0.0.1 <PREFIX>_INSTANCE_ROOT=<instanceRoot>`.
+3. Set `INSTANCE_SERVICE=<unit>` + `PORT=<port>` in the deploy environment.
+4. Front it with TLS and point the origins at the real domain (see [self-host.md](self-host.md)).