@rizom/ops 0.2.0-alpha.29 → 0.2.0-alpha.291

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (59) hide show
  1. package/README.md +39 -2
  2. package/dist/brains-ops.js +404 -314
  3. package/dist/cert-bootstrap.d.ts +3 -1
  4. package/dist/content-repo-ref.d.ts +10 -0
  5. package/dist/content-repo.d.ts +1 -0
  6. package/dist/deploy.js +99 -166
  7. package/dist/directory-sync-stress-system.d.ts +74 -0
  8. package/dist/directory-sync-stress.d.ts +106 -0
  9. package/dist/entries/deploy.d.ts +3 -2
  10. package/dist/images.d.ts +79 -0
  11. package/dist/index.d.ts +7 -0
  12. package/dist/index.js +479 -298
  13. package/dist/legacy-pilot-migration.d.ts +11 -0
  14. package/dist/load-registry.d.ts +57 -6
  15. package/dist/observed-status.d.ts +1 -1
  16. package/dist/origin-ca.d.ts +1 -1
  17. package/dist/parse-args.d.ts +2 -10
  18. package/dist/preview-domain.d.ts +9 -0
  19. package/dist/push-secrets.d.ts +2 -9
  20. package/dist/push-target.d.ts +1 -2
  21. package/dist/reconcile-dry-run.d.ts +9 -0
  22. package/dist/run-command.d.ts +12 -3
  23. package/dist/run-subprocess.d.ts +1 -6
  24. package/dist/schema.d.ts +114 -162
  25. package/dist/secrets-encrypt.d.ts +7 -13
  26. package/dist/ssh-key-bootstrap.d.ts +1 -26
  27. package/dist/stage-legacy-crossover.d.ts +20 -0
  28. package/dist/stress-command.d.ts +14 -0
  29. package/dist/stress-git-checkout.d.ts +21 -0
  30. package/dist/stress-health-monitor.d.ts +46 -0
  31. package/dist/upgrade.d.ts +10 -0
  32. package/dist/user-add.d.ts +15 -0
  33. package/dist/verify-user.d.ts +22 -0
  34. package/package.json +47 -42
  35. package/templates/rover-pilot/.env.schema +24 -3
  36. package/templates/rover-pilot/.github/actions/varlock-env/action.yml +47 -0
  37. package/templates/rover-pilot/.github/workflows/build.yml +70 -17
  38. package/templates/rover-pilot/.github/workflows/deploy.yml +74 -61
  39. package/templates/rover-pilot/.github/workflows/directory-sync-stress.yml +119 -0
  40. package/templates/rover-pilot/.github/workflows/reconcile.yml +10 -4
  41. package/templates/rover-pilot/.github/workflows/upgrade.yml +104 -0
  42. package/templates/rover-pilot/README.md +16 -6
  43. package/templates/rover-pilot/deploy/scripts/decrypt-user-secrets.ts +80 -24
  44. package/templates/rover-pilot/deploy/scripts/helpers.ts +3 -0
  45. package/templates/rover-pilot/deploy/scripts/install-health-watchdog.ts +144 -0
  46. package/templates/rover-pilot/deploy/scripts/resolve-deploy-handles.ts +12 -3
  47. package/templates/rover-pilot/deploy/scripts/resolve-missing-images.ts +13 -0
  48. package/templates/rover-pilot/deploy/scripts/resolve-user-config.ts +44 -9
  49. package/templates/rover-pilot/deploy/scripts/sync-content-repo.ts +51 -47
  50. package/templates/rover-pilot/deploy/scripts/update-dns.ts +14 -4
  51. package/templates/rover-pilot/deploy/scripts/validate-secrets.ts +12 -1
  52. package/templates/rover-pilot/docs/canonical-crossover-record.md +104 -0
  53. package/templates/rover-pilot/docs/onboarding-checklist.md +28 -17
  54. package/templates/rover-pilot/docs/operator-playbook.md +259 -29
  55. package/templates/rover-pilot/docs/user-onboarding.md +47 -462
  56. package/templates/rover-pilot/pilot.yaml +3 -4
  57. package/templates/rover-pilot/.kamal/hooks/pre-deploy +0 -9
  58. package/templates/rover-pilot/deploy/Dockerfile +0 -30
  59. package/templates/rover-pilot/deploy/kamal/deploy.yml +0 -40
@@ -0,0 +1,104 @@
1
+ # Canonical Crossover Approval Record
2
+
3
+ > Draft evidence record only. Completing this file does not authorize a merge, publish, reconcile, or deployment.
4
+
5
+ Do not include secret values, private keys, access tokens, or decrypted user configuration.
6
+
7
+ ## Authorization and freeze
8
+
9
+ - Operator:
10
+ - Approval reference:
11
+ - Approved window:
12
+ - Freeze start:
13
+ - Build workflow disabled and idle:
14
+ - Reconcile workflow disabled and idle:
15
+ - Deploy workflow disabled and idle:
16
+
17
+ ## Reviewed inputs
18
+
19
+ - `brains` crossover commit:
20
+ - Private-pilot source commit used for staging:
21
+ - Canonical review commit:
22
+ - Secret-free review diff SHA-256:
23
+ - Identity-review evidence SHA-256:
24
+ - Reviewed hosted-site pin manifest SHA-256:
25
+ - Source and review worktrees clean:
26
+
27
+ ## Hosted site and theme pins
28
+
29
+ List every hosted site. External site and theme package versions must be exact and must
30
+ match the staged user desired state; do not infer them from the brain or from each other.
31
+
32
+ | Handle | Site package | Exact site version | Theme package | Exact theme version | Package/image evidence |
33
+ | ------ | ------------ | ------------------ | ------------- | ------------------- | ---------------------- |
34
+ | | | | | | |
35
+
36
+ ## Forward artifact pins
37
+
38
+ | Artifact | Exact version | Registry integrity or digest | Verified installable |
39
+ | ---------------------- | ------------- | ---------------------------- | -------------------- |
40
+ | `@rizom/brain` | | | |
41
+ | `@rizom/ops` | | | |
42
+ | private-pilot lockfile | n/a | | |
43
+
44
+ Record every image. Tags alone are not immutable evidence.
45
+
46
+ | Cohort/handle | Config commit | Image repository | Image tag | Image digest |
47
+ | ------------- | ------------- | ---------------- | --------- | ------------ |
48
+ | | | | | |
49
+
50
+ ## Rollback pair
51
+
52
+ - Prior private-pilot commit:
53
+ - Prior `@rizom/ops` version and registry integrity:
54
+
55
+ | Cohort/handle | Prior config commit | Prior image repository | Prior image tag | Prior image digest |
56
+ | ------------- | ------------------- | ---------------------- | --------------- | ------------------ |
57
+ | | | | | |
58
+
59
+ ## Identity review
60
+
61
+ - Repository and image names unchanged or explicitly approved:
62
+ - GitHub organization and content repository identities unchanged:
63
+ - Server, domain, Cloudflare zone, and ATProto identities unchanged:
64
+ - Secret selector names and encrypted secret artifacts unchanged:
65
+ - Per-user runtime-version and site-package tag inputs reviewed:
66
+ - No plaintext source secrets copied into the review artifact:
67
+ - No removed model, preset, or old-format schema discriminator remains:
68
+ - Every external hosted site and theme package has an exact reviewed pin:
69
+
70
+ ## Offline convergence
71
+
72
+ Command:
73
+
74
+ ```sh
75
+ bunx brains-ops reconcile-all <canonical-review-copy> --dry-run
76
+ ```
77
+
78
+ - First-pass reconciler-owned changed files:
79
+ - Second-pass reconciler-owned changed files (must be zero):
80
+ - Observational `views/users.md` unchanged by reconciliation:
81
+ - Review copy unchanged:
82
+ - External content-repository access blocked:
83
+
84
+ ## Pre-window validation
85
+
86
+ - Package tests, typecheck, lint, build, and packed-consumer startup:
87
+ - Architecture and dependency boundaries:
88
+ - Environment-schema and workspace checks:
89
+ - Crossover migration and comment preservation:
90
+ - Canary-first order and health checks reviewed:
91
+ - Paired rollback reviewed:
92
+
93
+ ## Execution record
94
+
95
+ Complete only inside the explicitly approved maintenance window.
96
+
97
+ - Unified packages published and verified:
98
+ - Canonical desired state committed while automation remained frozen:
99
+ - Image digest set matched this record:
100
+ - First reconcile reviewed:
101
+ - Per-instance health, MCP authorization, identity, content, and site checks:
102
+ - Second reconcile produced zero reconciler-owned drift and no deploy work:
103
+ - Freeze lifted:
104
+ - One-week soak start:
@@ -4,28 +4,39 @@
4
4
  2. Run `bunx brains-ops age-key:bootstrap <repo> --push-to gh`.
5
5
  3. Fill in `pilot.yaml`.
6
6
  - keep your pinned `brainVersion`
7
- - confirm shared selectors for `aiApiKey`, `gitSyncToken`, and `mcpAuthToken`
7
+ - confirm shared selectors for `aiApiKey`, `gitSyncToken`, and `contentRepoAdminToken`
8
+ - use different tokens for `contentRepoAdminToken` and `gitSyncToken`: admin creates/checks content repos; sync is used by runtime directory-sync
8
9
  - confirm `agePublicKey`
9
- 4. Add or edit `users/<handle>.yaml`.
10
- - Discord is enabled by default for pilot users
11
- - if the user should be an anchor there, set `discord.anchorUserId` to their Discord user ID
12
- 5. Add the user to a cohort in `cohorts/*.yaml`.
10
+ 4. Run `bunx brains-ops user:add <repo> <handle> --cohort <cohort>`.
11
+ - Web chat is the primary interface; it needs no per-user setup beyond the passkey.
12
+ - `user:add` currently writes `discord: enabled: true`; set it to `false` unless the user's cohort actually uses Discord.
13
+ - if the user should be an anchor on Discord, add `--anchor-id <discord-user-id>`.
14
+ - the command creates `users/<handle>.yaml`, `users/<handle>.secrets.yaml`, and the cohort membership without duplicating existing entries.
15
+ 5. Edit the generated user file if the anchor profile needs richer metadata.
16
+ - Set `setup.delivery: email` and `setup.email` so the user gets the passkey setup email — this is the default onboarding path.
17
+ - For ATProto publishing, add `atproto.identifier` to the user file; put only `atprotoAppPassword` in the per-user secrets file.
18
+ - Ensure `SETUP_EMAIL_API_KEY` and `SETUP_EMAIL_FROM` exist as GitHub Secrets before deploying any email-setup user.
13
19
  6. Run `bunx brains-ops render <repo>`.
14
20
  7. Run `bunx brains-ops ssh-key:bootstrap <repo> --push-to gh`.
15
21
  8. Run `bunx brains-ops cert:bootstrap <repo> --push-to gh`.
16
- 9. Keep raw user secret material locally for now (`.env.local`, file-backed env vars, or equivalent local inputs).
22
+ 9. Keep raw user secret material locally for now (`.env.local`, file-backed env vars, or equivalent local inputs), including `CONTENT_REPO_ADMIN_TOKEN` for operator onboarding.
17
23
  10. Run `bunx brains-ops secrets:encrypt <repo> <handle>`.
18
24
  11. Commit and push `users/<handle>.secrets.yaml.age`.
19
25
  12. Run `bunx brains-ops onboard <repo> <handle>`.
20
- 13. Verify the deployed rover core contract:
21
- - `https://<handle>.rizom.ai/health` returns `200`
22
- - unauthenticated `POST https://<handle>.rizom.ai/mcp` returns `401`
23
- 14. For fleet upgrades, edit `pilot.yaml.brainVersion` and push once; CI rebuilds the shared image tag, refreshes generated user env files, and redeploys affected users.
24
- 15. Hand the Discord setup details to the user.
25
- 16. Hand over the browser defaults:
26
+ 13. Verify the deployed canonical contract:
27
+ - `https://<handle>.rizom.ai/health/operate` returns `200`
28
+ - `https://<handle>.rizom.ai/chat` loads the web chat and accepts passkey sign-in
29
+ - unauthenticated `POST https://<handle>.rizom.ai/mcp` returns the expected auth failure
30
+ - content repo exists and runtime sync is healthy
31
+ - background jobs are not repeatedly failing, except for expected missing optional integrations
32
+ - when `site` is selected, the browser/CMS surfaces load and the initial app-managed site build completes
33
+ 14. For fleet upgrades, edit `pilot.yaml.brainVersion` and push once; CI rebuilds the required default/site image tags, refreshes generated user env files, and redeploys affected users. Every external site and theme package keeps its own required exact version pin and never follows the brain version implicitly.
34
+ 15. Confirm the user received the setup email, registered their passkey, and can sign in to web chat at `https://<handle>.rizom.ai/chat`. That completes the default onboarding; everything below is per-cohort extras.
35
+ 16. Hand over the browser surfaces:
36
+ - Chat (primary): `https://<handle>.rizom.ai/chat`
26
37
  - Dashboard: `https://<handle>.rizom.ai/`
27
- - CMS: `https://<handle>.rizom.ai/cms`
28
- - GitHub token guidance for CMS access to the user's private content repo
29
- 17. If they need direct client access, also hand over the MCP connection details.
30
- 18. If you are also giving them a content repo workflow, describe it as optional and frame git/Obsidian as an advanced file-based path, not the default.
31
- 19. Send `docs/user-onboarding.md` to the user as the pilot handoff guide.
38
+ - CMS: `https://<handle>.rizom.ai/cms`, plus GitHub token guidance if CMS editing is part of their cohort
39
+ 17. For Discord-enabled cohorts, hand the Discord setup details to the user as a secondary chat surface.
40
+ 18. If they need direct client access (MCP), use OAuth/passkey-capable clients where possible.
41
+ 19. If you are also giving them a content repo workflow, describe it as optional and frame git/Obsidian as an advanced file-based path, not the default.
42
+ 20. Send `docs/user-onboarding.md` to the user as the pilot handoff guide.
@@ -9,30 +9,97 @@ Treat these as checked-in deploy artifacts in the pilot repo:
9
9
  - `deploy/scripts/`
10
10
  - `.github/workflows/build.yml`
11
11
  - `.github/workflows/deploy.yml`
12
+ - `.github/workflows/directory-sync-stress.yml`
12
13
  - `.github/workflows/reconcile.yml`
13
14
 
14
15
  `.env.schema` is the single source of truth for required and sensitive deploy vars.
15
16
  The deploy scripts and workflows should read from that contract instead of inventing a second list.
16
17
 
17
- The shared pilot image tag is `brain-${brainVersion}`:
18
+ The default pilot image tag is `brain-${brainVersion}`:
18
19
 
19
- - build publishes `brain-${brainVersion}`
20
+ - build publishes `brain-${brainVersion}` for users without a site override
21
+ - a site override gets an isolated `brain-${brainVersion}-sites-${packageHash}` image
20
22
  - generated `users/<handle>/.env` carries `BRAIN_VERSION=<brainVersion>`
21
- - deploy sets `VERSION=brain-${brainVersion}`
23
+ - build and deploy derive the same effective image tag from the resolved registry
22
24
 
23
25
  ## Version bump flow
24
26
 
25
27
  When `pilot.yaml.brainVersion` changes and you push:
26
28
 
27
- 1. build publishes the new shared image tag
29
+ 1. build publishes the new default image and any required site images
28
30
  2. reconcile refreshes generated `users/<handle>/.env`
29
31
  3. deploy runs for handles whose generated config changed
30
32
  4. generated file commits happen once in a final aggregation step after the deploy matrix finishes
31
33
 
34
+ Every external site and theme package has its own exact version pin. A cohort or
35
+ pilot brain-version bump never changes those package versions implicitly; update each
36
+ pin deliberately from reviewed package and image evidence.
37
+
32
38
  When a push changes only deploy contract files and no generated `users/<handle>/.env` or `users/<handle>/brain.yaml` files, the deploy workflow exits through its explicit no-op path and prints `No affected user configs; skipping deploy.`
33
39
 
34
40
  They are scaffolded from `@rizom/ops`, then versioned in this repo like any other deploy contract.
35
41
 
42
+ ## Canonical contract crossover maintenance window
43
+
44
+ Do not run this procedure without explicit operator approval. The canonical desired state, canonical `@rizom/ops`, and unified runtime image form one contract and must move or roll back together. Complete `docs/canonical-crossover-record.md` as the approval evidence without adding secret values.
45
+
46
+ Before the window, record and review:
47
+
48
+ - the prior pilot commit and exact `@rizom/ops` version;
49
+ - every prior runtime image tag and immutable digest;
50
+ - the reviewed canonical pilot commit;
51
+ - the exact unified `@rizom/brain` and `@rizom/ops` versions;
52
+ - every unified image tag and immutable digest;
53
+ - canary-first rollout order, followed by the remaining cohorts;
54
+ - expected `/health/operate` version, unauthenticated MCP response, site marker, and content repository identity for each posture.
55
+
56
+ Run `bunx brains-ops reconcile-all <canonical-review-copy> --dry-run` against the isolated review copy. The command blocks external content-repository access, leaves the review copy untouched, lists both passes' changed files, and must report second-pass zero drift. Reconciliation owns generated per-user config; only `render` owns the observational `views/users.md` projection.
57
+
58
+ During the approved window:
59
+
60
+ 1. Freeze unrelated merges and releases. Wait for active Build, Reconcile, and Deploy runs to finish, then disable all three pilot workflows with `gh workflow disable build.yml`, `gh workflow disable reconcile.yml`, and `gh workflow disable deploy.yml`.
61
+ 2. Publish and verify the reviewed unified runtime and matching ops artifacts. Do not update pilot desired state until the exact versions are installable and the expected images can be built.
62
+ 3. Apply the reviewed canonical pilot revision while automation remains disabled. Confirm repository names, server/domain identity, content repositories, secret selectors, image names, and tag identity against the review diff.
63
+ 4. Enable only Build, run it for the canonical desired-state revision, and record every resulting image digest. Stop if the observed digest set differs from the cutover record.
64
+ 5. Enable only Reconcile, run it once, and review its generated per-user config commit. It must not rewrite `views/users.md`, and no generated file may combine canonical config with a retired image version.
65
+ 6. Enable Deploy and deploy one handle at a time in the approved order. After each deploy, run `bunx brains-ops verify-user . <handle>`, render observed fleet status, and complete the manual identity, content-sync, and app-managed site checks.
66
+ 7. Run Reconcile a second time. Require no reconciler-owned generated diff and no deploy work before re-enabling normal automation and lifting the merge/release freeze. Observed status rendering remains separate from this convergence gate.
67
+
68
+ If any gate fails, disable all three workflows again. Restore the prior pilot desired-state and dependency revision, reconcile with the prior ops version, and redeploy the prior image tag/digest as one rollback pair. Verify the prior `/health/operate` version and identity/content/site checks before re-enabling automation. Never restore only config or only an image.
69
+
70
+ ## Directory-sync stress gate
71
+
72
+ Use the manual `Directory Sync Stress` workflow only against a disposable smoke user. It refuses a target unless the handle, domain, and content repository all identify smoke, the confirmation input exactly matches `stress:<handle>`, and the user desired state declares the hermetic posture below. Reconcile and deploy this posture before running the workload:
73
+
74
+ ```yaml
75
+ embeddingEnabled: false
76
+ topicExtractionEnabled: false
77
+ ```
78
+
79
+ Before authorizing a workload, dispatch the workflow once with `verify_only: true`. That mode loads the same Bitwarden/Varlock content credential, clones the smoke content repository, and runs `git push --dry-run` against a temporary stress ref. It creates no ref, performs no content write, does not contact the deployed runtime, and skips cleanup because no probes were created.
80
+
81
+ Profiles are deterministic and reversible:
82
+
83
+ - `regression`: 20 probes;
84
+ - `load`: ramps to 350 probes, updates all, renames 100, updates again, then deletes all;
85
+ - `stress`: ramps to 700 probes and renames 200 before cleanup.
86
+
87
+ The workflow loads operator credentials through Bitwarden/Varlock, but it is separate from Deploy and cannot deploy an image. It creates a rollback branch before the first content write, gates on health timeouts, watchdog restarts, and external AI usage during the monitored workload window, preserves warmup and cleanup samples as evidence, uploads JSON/Markdown/runtime artifacts, and runs an independent idempotent cleanup job with `if: always()`. Once cleanup confirms that no probes remain, it also prunes retained `ops/directory-sync-stress-backup-*` branches; if probes remain, the branches stay available for recovery.
88
+
89
+ Treat any gated health failure, restart, OOM, residual probe, or entity-baseline drift as a failed gate. Do not restart the target during measurement. Recovery is a separate operator action after evidence collection.
90
+
91
+ ## Stale deploy lock recovery
92
+
93
+ Kamal intentionally leaves its remote deploy lock in place when a deployment is cancelled or interrupted. Confirm that no deployment for the user is still active before releasing the lock, then use the deploy workflow's explicit recovery input:
94
+
95
+ ```sh
96
+ gh workflow run Deploy --ref main \
97
+ -f handle=<handle> \
98
+ -f release_stale_lock=true
99
+ ```
100
+
101
+ Recovery is opt-in and scoped to one handle. Normal push, reconcile, and manual deploy runs never remove a lock automatically.
102
+
36
103
  ## Bootstrap flow
37
104
 
38
105
  For this fleet, operator-local secret material remains the source of truth during onboarding and rotation. The repo stores encrypted per-user secrets, not raw values.
@@ -53,51 +120,214 @@ Preview hosts use the shape `<handle>-preview.rizom.ai`, so one wildcard origin
53
120
 
54
121
  ## Upgrading operator behavior
55
122
 
56
- When `@rizom/ops` changes the scaffolded deploy contract:
123
+ The pilot repository pins `@rizom/ops` in `package.json`. The scheduled and manually dispatched Upgrade workflow owns routine upgrades to that pin. It refreshes the scaffold on a branch and opens a reviewable PR; it does not change runtime desired state or authorize a deployment.
124
+
125
+ Because scaffold refreshes can update `.github/workflows/*`, the workflow must not push with its Actions `GITHUB_TOKEN`. Configure a dedicated GitHub App:
126
+
127
+ 1. Install it only on this pilot repository.
128
+ 2. Grant repository permissions `Contents: Read and write`, `Pull requests: Read and write`, and `Workflows: Read and write`; grant nothing else.
129
+ 3. Store its App ID as the repository Actions variable `OPS_UPGRADE_APP_ID`.
130
+ 4. Store its private key as the repository Actions secret `OPS_UPGRADE_APP_PRIVATE_KEY`.
131
+
132
+ The workflow explicitly requests only those three permissions. Checkout persists no credential. After the freshly published `@rizom/ops` finishes, the workflow checks whether it produced a change; only then does it mint a short-lived, repository-scoped App token for the push-and-open-PR steps. A credential that can rewrite `.github/workflows/` therefore does not exist while upgraded package code runs.
133
+
134
+ If `OPS_UPGRADE_APP_ID` is unset, or token creation, branch push, or PR creation fails, the run stops with a non-zero status. Repair the CI credential path; never fall back to an operator's personal SSH key or token.
135
+
136
+ Adopting this credential flow in an existing pilot repository requires one explicitly reviewed bootstrap PR because the old Upgrade workflow cannot update itself. After that merge, routine upgrades run entirely in CI:
137
+
138
+ 1. dispatch Upgrade with an exact version, or let its schedule select `latest`;
139
+ 2. review the generated package, lockfile, deploy-script, and workflow diff;
140
+ 3. merge the upgrade PR only after its checks pass;
141
+ 4. change runtime desired state separately through the approved canary or fleet rollout flow.
142
+
143
+ ## Canonical verification notes
144
+
145
+ Use the verification script after deploy:
146
+
147
+ ```sh
148
+ bunx brains-ops verify-user . <handle>
149
+ ```
150
+
151
+ For every bundle posture it checks:
152
+
153
+ - `https://<handle>.rizom.ai/health/operate` returns `200`;
154
+ - unauthenticated `POST https://<handle>.rizom.ai/mcp` returns the expected auth failure;
155
+ - background jobs are not repeatedly failing, except for missing optional integrations.
156
+
157
+ A `core`-only instance is MCP-only; a bare `GET /` may return `401` without indicating a bad deploy. When `site` is selected, verification also checks the browser and CMS/login surfaces.
158
+
159
+ Manual checks that remain:
160
+
161
+ - initial app-managed site output is correct for the expected content/theme;
162
+ - content repository identity and runtime sync are healthy;
163
+ - passkey setup/handoff is completed from the setup email.
164
+
165
+ ## One-user canonical site canary
57
166
 
58
- 1. bump `@rizom/ops` in `package.json`
59
- 2. rerun the relevant scaffold/reconcile flow
60
- 3. review the resulting changes to `.env.schema`, `deploy/scripts/`, and workflows in git
61
- 4. commit the updated deploy artifacts together
167
+ Run this before adding custom site/theme packages or rolling a larger browser/CMS-first cohort.
62
168
 
63
- ## Rover-core verification notes
169
+ 1. Create or choose a canary cohort with explicit bundles:
64
170
 
65
- Rover core is MCP-only. Do not expect the bare domain to serve a website.
171
+ ```yaml
172
+ bundles:
173
+ - core
174
+ - site
175
+ - publishing
176
+ ```
66
177
 
67
- Use these checks after deploy:
178
+ 2. Add exactly one canary user to that cohort.
179
+ 3. For browser/CMS-first onboarding, configure setup email in `users/<handle>.yaml`:
68
180
 
69
- - `https://<handle>.rizom.ai/health` should return `200`
70
- - unauthenticated `POST https://<handle>.rizom.ai/mcp` should return `401 Unauthorized: Bearer token required`
71
- - a bare `GET /` may also return `401`; that is expected for rover core and does not indicate a bad deploy
181
+ ```yaml
182
+ setup:
183
+ delivery: email
184
+ email: user@example.com
185
+ ```
72
186
 
73
- ## Discord bot token checklist
187
+ 4. Encrypt the user's secrets and commit only the `.age` file.
188
+ 5. Run `bunx brains-ops onboard . <handle>`.
189
+ 6. Run `bunx brains-ops verify-user . <handle>` with no custom site/theme overrides.
190
+ 7. Ask the user to complete passkey setup from the setup email.
191
+ 8. Continue to visual customization only after the canary is healthy.
192
+
193
+ Rollback must restore the prior desired-state revision and prior runtime image together. Never pair canonical config with the retired image, or retired config with the canonical image.
194
+
195
+ ## Hosted site and theme package contract
196
+
197
+ Start with the public [site mockup migration guide](https://github.com/rizom-ai/brains/blob/main/docs/site-mockup-migration.md), then apply these hosted-fleet requirements:
198
+
199
+ - A site package must default-export `defineSite(...)` and import its authoring API only from `@rizom/site`.
200
+ - A theme package must default-export its CSS as a string. Hosted custom themes currently use the `@rizom/*` scope so the fleet image installs them with the site package; `@brains/*` themes are bundled with `@rizom/brain`.
201
+ - Site and custom theme packages must be public npm packages that install without registry credentials.
202
+ - Site, theme, and brain packages publish independently. Hosted configuration requires exact site and external-theme version pins and never derives one package version from another.
203
+ - Keep site structure and theme CSS in separate packages. Do not put private content or secrets in either package.
204
+
205
+ Configure a user in `users/<handle>.yaml`:
206
+
207
+ ```yaml
208
+ siteOverride:
209
+ package: "@rizom/site-example"
210
+ version: <exact-site-version>
211
+ theme: "@rizom/theme-example"
212
+ themeVersion: <exact-theme-version>
213
+ ```
214
+
215
+ Missing external package versions fail desired-state validation. A site override
216
+ produces an isolated per-instance image; it never changes the fleet's shared default
217
+ image. Bundled `@brains/*` themes omit `themeVersion` because they are not installed as
218
+ separate packages.
219
+
220
+ ### Custom-package canary and rollback
221
+
222
+ 1. Confirm the exact site/theme versions are public-installable without npm credentials.
223
+ 2. Apply the exact package names and versions to one healthy canonical site canary.
224
+ 3. Reconcile the canary, push the generated output, and let build/deploy create its site image.
225
+ 4. Run `bunx brains-ops verify-user . <handle>`.
226
+ 5. Manually verify the site, theme, CMS, content sync, and passkey sign-in before adding more users.
227
+
228
+ To roll back, remove or change `siteOverride`, reconcile, and redeploy that user.
229
+ The default image and other users remain untouched.
230
+
231
+ ## Setup email checklist
232
+
233
+ Use this for browser/CMS-first users who should receive their own first-passkey setup link by email.
234
+
235
+ 1. Add setup delivery to the user file:
236
+
237
+ ```yaml
238
+ setup:
239
+ delivery: email
240
+ email: user@example.com
241
+ ```
242
+
243
+ 2. Configure these GitHub Secrets before deploy:
244
+ - `SETUP_EMAIL_API_KEY`
245
+ - `SETUP_EMAIL_FROM`
246
+
247
+ 3. Reconcile/deploy the user or cohort:
248
+ - `bunx brains-ops onboard . <handle>`
249
+ - or `bunx brains-ops reconcile-cohort . <cohort>`
250
+
251
+ 4. Verify the generated `users/<handle>/brain.yaml` contains `auth-service.setupEmail` and `email` interface config.
252
+ 5. Ask the user to complete passkey setup from the email link, then use:
253
+ - Dashboard: `https://<handle>.rizom.ai/`
254
+ - CMS: `https://<handle>.rizom.ai/cms`
255
+
256
+ Notes:
257
+
258
+ - The setup URL is generated and sent by the running brain; operators should not scrape logs or SSH into the instance to retrieve it.
259
+ - The auth service owns setup email dedupe. It should not resend for the same persisted setup token after restart, but should retry failed delivery and resend after token rotation.
260
+ - `SETUP_EMAIL_FROM` is not marked required because fleets without email setup can omit it, but it is required for users with `setup.delivery: email`.
261
+
262
+ ## AT Protocol smoke/config checklist
263
+
264
+ Use this when enabling AT Protocol publishing for a single pilot user.
265
+
266
+ 1. Add the public PDS identifier to the user file. Prefer the account DID as
267
+ the identifier — it survives handle changes. Add `accountDid` too when the
268
+ member wants their handle verified against their subdomain
269
+ (`@<handle>.<domainSuffix>`): the brain then serves it at
270
+ `/.well-known/atproto-did` and Bluesky's "I have my own domain" HTTP
271
+ verification passes with no DNS records.
272
+
273
+ ```yaml
274
+ atproto:
275
+ identifier: did:plc:example123
276
+ accountDid: did:plc:example123
277
+ ```
278
+
279
+ Only for the PDS account designated by the protocol authority's `_lexicon`
280
+ DNS TXT record, also set `lexiconAuthority: true`. Every other fleet user
281
+ must omit it.
282
+
283
+ 2. Put the app password in `users/<handle>.secrets.yaml`:
284
+
285
+ ```yaml
286
+ atprotoAppPassword: <app-password>
287
+ ```
288
+
289
+ 3. Encrypt the per-user secret payload:
290
+ - `bunx brains-ops secrets:encrypt . <handle>`
291
+ 4. Reconcile/deploy the user or cohort:
292
+ - `bunx brains-ops onboard . <handle>`
293
+ - or `bunx brains-ops reconcile-cohort . <cohort>`
294
+ 5. Verify the generated `users/<handle>/brain.yaml` contains `plugins.atproto.identifier` (plus `accountDid` and `lexiconAuthority` when configured) and `appPassword: ${ATPROTO_APP_PASSWORD}`.
295
+
296
+ Notes:
297
+
298
+ - The ATProto identifier and authority flag are public instance config and belong in `users/<handle>.yaml`. Only the DNS-designated authority account may set `lexiconAuthority: true`.
299
+ - The ATProto app password is secret and belongs only in the encrypted per-user secret payload.
300
+ - For smoke deployments, pin only the smoke cohort/user to the released brain version that contains ATProto support.
301
+
302
+ ## Discord application credential checklist
74
303
 
75
304
  Use this when enabling Discord for a pilot user.
76
305
 
77
306
  1. Pick the user handle (for example `smoke`).
78
307
  2. Open the Discord Developer Portal.
79
- 3. Create a **new application** for that user's rover.
308
+ 3. Create a **new application** for that user's brain.
80
309
  4. Add a **Bot** to the application.
81
- 5. Copy the bot token.
82
- 6. Put that value in `.env` or `.env.local` in this repo as `DISCORD_BOT_TOKEN=...` while onboarding that user.
310
+ 5. Copy the bot token, application public key, and application ID.
311
+ 6. Put those values in `.env` or `.env.local` while onboarding that user:
312
+ - `DISCORD_BOT_TOKEN=...`
313
+ - `DISCORD_PUBLIC_KEY=...`
314
+ - `DISCORD_APPLICATION_ID=...`
83
315
  7. Keep `discord.enabled: true` in `users/<handle>.yaml` unless you explicitly want to disable the primary pilot interface.
84
- 8. Encrypt the current per-user secret payload:
316
+ 8. Encrypt the current per-user credential payload:
85
317
  - `bunx brains-ops secrets:encrypt . <handle>`
86
318
  9. Reconcile/deploy the user or cohort:
87
-
88
- - `bunx brains-ops onboard . <handle>`
89
- - or `bunx brains-ops reconcile-cohort . <cohort>`
90
-
91
- 11. In the Discord Developer Portal, generate an install URL and invite the bot to the right server.
92
- 12. Send a test message in Discord and confirm the rover responds.
319
+ - `bunx brains-ops onboard . <handle>`
320
+ - or `bunx brains-ops reconcile-cohort . <cohort>`
321
+ 10. In the Discord Developer Portal, generate an install URL and invite the bot to the right server.
322
+ 11. Send a test message in Discord and confirm the brain responds.
93
323
 
94
324
  Notes:
95
325
 
96
- - Use **one bot token per user/rover**.
97
- - Do not reuse the same Discord bot token across multiple pilot users.
326
+ - Use **one Discord application credential set per user/brain**.
327
+ - Do not reuse the same Discord application across multiple pilot users.
98
328
  - Discord is the default pilot interface moving forward.
99
329
  - The encrypted `users/<handle>.secrets.yaml.age` file is the durable checked-in deploy input; your local env is only the operator staging source.
100
- - MCP is optional and mainly for direct client access or specific testing workflows.
330
+ - Direct MCP client access should use OAuth/passkey-capable clients where possible.
101
331
  - When explaining the content workflow, describe it first as a normal **git repo** of **markdown/text files**.
102
332
  - Position **Obsidian** as optional: it is just one possible editor for those same files, not the default requirement.
103
333