@mediamonks/canopy-cli 0.7.20 → 0.7.21

package/dist/docs/infrastructure.md

@@ -1,406 +0,0 @@
----
-title: Infrastructure as code
-description: Terraform for the account-level Cloudflare pieces Canopy needs.
----
-
-Canopy's Cloudflare footprint is small on purpose: almost everything
-per-project is provisioned at runtime by canopy-api (`apps/api/src/cf.ts`,
-reached through the Cloudflare `PlatformAdapter` in
-`apps/api/src/platform/cloudflare.ts`), so the only things worth managing
-declaratively are a handful of platform-wide resources. Those live in
-[`infra/`](../../infra) as OpenTofu; the rest is bootstrap scripts and this
-runbook.
-
-**Live platform (2026-08-01):** Monks Explore account
-`1c826c7b063fd9ca5e1b1bdaabd05feb`, zone `canopy.website`. The steps below
-are the verified path used for that move (personal account
-`9513cbfa5b182546f737501b10e1ed85` / `thanarie.dev` → company). Re-run them
-to stand the platform up in another fresh account, or to rebuild after a
-wipe.
-
-Scope of a fresh stand-up:
-
-1. the **platform workers** — `canopy-api` and `canopy-gateway` — deployed and
-   healthy (these aren't optional; nothing works without them), and
-2. the **four core projects** — `dashboard`, `www`, `docs`, `kitchen-sink` —
-   re-created and deployed _on_ the platform like any other project.
-
-Everyone else's projects are **not** carried over — a new account starts with
-an empty control-plane registry (see [Non-goals](#non-goals)).
-
-See decision **D14** in [`02-decisions.md`](../02-decisions.md) for the why.
-
-## What's code vs. what's clicked
-
-**OpenTofu** (`infra/*.tf`): zone, proxied wildcard DNS record, control-plane
-D1, R2 bucket, AI Gateway shell, optional ACM cert pack.
-
-**wrangler** (`apps/{api,gateway}/wrangler.{main,staging}.jsonc`): the two
-platform Worker scripts, their bindings, and the `*.<zone>/*` route — one
-config file per Cloudflare site (same names as CLI profiles). Deployed with
-`pnpm run deploy:main` / `deploy:staging`, not Tofu. Plain `wrangler.jsonc`
-is local `wrangler dev` only.
-
-**`canopy deploy`** (`canopy.json` in each project): the core projects
-(`dashboard`, `www`, `docs`, `kitchen-sink`) — ordinary platform projects
-that dogfood the product, not infrastructure.
-
-**Bootstrap scripts** (`infra/scripts/`): runtime API tokens, worker secrets,
-AI Gateway BYOK keys + spend rule.
-
-**Manual / billing (clicked once in the dashboard):** create the account &
-attach it to the org; add the domain; mint bootstrap API token(s); optionally
-enable paid Workers products. Plus OneLogin: redirect URIs for the new host
-(if they aren't already set).
-
-## Non-goals
-
-- **No project-data migration.** Per-project D1s, R2 blobs, and deploy history
-  stay in the old account. The new control-plane D1 is empty; the core
-  projects are re-created from scratch (they hold no precious data — `www`,
-  `dashboard`, and `docs` are stateless, `kitchen-sink` is a demo), and everyone else's
-  projects are re-created as needed. (If that ever changes, the export/import
-  primitives already exist in `apps/api/src/cf.ts` and `canopy db snapshot|restore`.)
-- **No Tofu management of Workers/secrets/per-project resources** — see the
-  table in [`infra/README.md`](../../infra/README.md).
-- **No per-project Workers on the Cloudflare target today.** The CF
-  `PlatformAdapter` reports `capabilities.workers = false` (and
-  schedules/secrets likewise). Project backends, schedules, and project
-  secrets live on targets that support them (e.g. GCP). Kitchen-sink's
-  `worker` field in `canopy.json` must be omitted for a CF deploy — see
-  [step 10](#10-log-in-grant-yourself-admin-then-deploy-the-core-projects).
-
----
-
-## Rollout — step by step
-
-Prereqs on your machine: `tofu` (or `terraform`), `wrangler` (via `pnpm` in
-the repo), `curl`, `jq`, `openssl`. Confirm wrangler can see the target
-account after `wrangler login`:
-
-```bash
-pnpm --filter api exec wrangler whoami
-# expect the new account listed (e.g. "Monks Explore")
-export CLOUDFLARE_ACCOUNT_ID="<account_id>"   # required when you belong to >1 account
-```
-
-### 1. Account & billing (dashboard)
-
-1. Create / choose the new account; attach it to the company org; ensure your
-   user is a member with rights to manage Workers / D1 / R2 / DNS.
-2. Add `canopy.website` to the account and repoint its nameservers at
-   Cloudflare. Wait for the zone to go **Active**. Note the **Account ID**
-   (`tofu` var `account_id`) and the **Zone ID**.
-3. **Delete any auto-created `www` record.** Cloudflare's onboarding scan often
-   adds a `www` A/CNAME. An explicit record beats the wildcard, and if it's
-   DNS-only (grey cloud) the request never reaches Workers — which would break
-   the `www` core project. The wildcard `*` alone must serve `www.canopy.website`.
-   (You can confirm via API once the bootstrap token exists — an empty zone is
-   fine.)
-4. **R2** must work (creating a probe bucket is enough to confirm). **Workers
-   Paid + Workers for Platforms** are required for per-project workers
-   (Phase 2 item 1, D18): once the subscriptions are active, run
-   `infra/scripts/create-dispatch-namespace.sh` to create the
-   `canopy-projects` dispatch namespace (the `dispatch_namespaces` bindings in
-   `apps/api/wrangler.main.jsonc` and `apps/gateway/wrangler.main.jsonc` fail
-   to deploy without it), and make sure `CF_API_TOKEN` carries **Workers
-   Scripts Write** (re-mint with `create-api-tokens.sh` if it predates this).
-   A site without the binding still runs — workers/schedules/secrets simply
-   answer 501. **No Advanced Certificate Manager** — `*.canopy.website`
-   is a single-level wildcard covered by free Universal SSL. (ACM would only
-   be needed if nesting as `*.canopy.<some-other-domain>`.)
-
-> **Apex is not covered.** `*` matches only one label, so `canopy.website`
-> itself isn't served by the gateway (by design — see
-> `apps/gateway/wrangler.main.jsonc`).
-> Decide separately whether the apex should redirect to `www.canopy.website`
-> (a Cloudflare Redirect Rule, cheap to add later); nothing in this runbook
-> depends on it.
-
-### 2. Bootstrap API token(s) (dashboard)
-
-You need credentials that can (a) run OpenTofu + wrangler ops, and (b) mint
-the least-privilege runtime tokens. In practice that is often **two**
-account-owned tokens — a single token that mixes zone DNS with **Account API
-Tokens · Write** has been flaky when edited in place.
-
-**Token A — `canopy-bootstrap` (Tofu + deploy ops)**
-
-| Resource                                                     | Permissions                                                             |
-| ------------------------------------------------------------ | ----------------------------------------------------------------------- |
-| Account                                                      | D1 Edit, Workers R2 Storage Edit, AI Gateway Edit, Workers Scripts Edit |
-| Zone `canopy.website` (nested under the account — see below) | Zone Edit, DNS Edit, Workers Routes Edit                                |
-
-**Token B — mint token (for `create-api-tokens.sh` only)**
-
-| Resource | Permissions                               |
-| -------- | ----------------------------------------- |
-| Account  | **Account API Tokens · Write** (and Read) |
-
-**Pitfalls we hit:**
-
-- The dashboard's "DNS" picks are easy to get wrong. You need zone-scoped
-  **DNS Write** / **Zone Write**, _not_ Account DNS Settings, DNS Views,
-  DNS Firewall, or Registrar Domains.
-- For **account-owned** tokens, zone resources must be **nested under the
-  account** in the policy JSON shape
-  `com.cloudflare.api.account.<account_id> → com.cloudflare.api.account.zone.<zone_id>`.
-  A top-level zone resource can look correct in the UI and still 10000-auth on
-  `/zones/:id/dns_records`.
-- `/user/tokens/verify` returns "Invalid API Token" for account-owned tokens —
-  use `GET /accounts/<account_id>/tokens/verify` instead.
-- Wrangler **OAuth** can list/create D1 and R2 and deploy Workers, but it
-  cannot manage DNS (and often cannot call AI Gateway). Don't rely on OAuth
-  alone for step 3.
-
-Store tokens outside git. Convenience pattern used in the move:
-
-```bash
-cd infra
-# gitignored via infra/.gitignore (.env / .env.*)
-cat > .env <<'EOF'
-CLOUDFLARE_API_TOKEN="<token A — canopy-bootstrap>"
-CLOUDFLARE_TOKEN_MINT_TOKEN="<token B — API Tokens Write>"
-EOF
-set -a && source .env && set +a
-```
-
-### 3. Provision static infra (OpenTofu)
-
-```bash
-cd infra
-cp terraform.tfvars.example terraform.tfvars     # set account_id + zone_name
-set -a && source .env && set +a                  # CLOUDFLARE_API_TOKEN = token A
-tofu init
-tofu import cloudflare_zone.platform <zone_id>   # adopt the domain from step 1
-tofu plan                                        # review — should be creates only
-tofu apply
-tofu output wrangler_patch_hint
-```
-
-Expect four creates: wildcard DNS `*`, D1 `canopy-control`, R2 `canopy-assets`,
-AI Gateway `canopy`. (If the AI Gateway resource fails schema validation,
-check `cache_invalidate_on_update` and friends against the pinned
-`cloudflare` provider minor — it shifted in ~> 5.22.)
-
-The R2 remote-state backend in `versions.tf` is commented out; for a solo
-stand-up, local state is fine. If you want shared state, create a
-`canopy-tfstate` bucket first, then uncomment and `tofu init -migrate-state`.
-
-### 4. Point the apps at the new resources
-
-Edit the **site** wrangler configs (not the local `wrangler.jsonc`) using
-`tofu output wrangler_patch_hint`. For production that is
-`wrangler.main.jsonc` in both apps; for the personal/staging account use
-`wrangler.staging.jsonc`. Keep both files in git — never comment-flip a
-single file to switch accounts.
-
-**`apps/api/wrangler.<site>.jsonc`:**
-
-- `account_id` / `vars.CF_ACCOUNT_ID` → account id
-- `d1_databases[0].database_id` → control-plane D1 UUID
-- `vars.AI_GATEWAY_ID` → unchanged (`canopy`) unless renamed
-- `r2_buckets[0].bucket_name` → unchanged (`canopy-assets`) unless renamed
-
-**`apps/gateway/wrangler.<site>.jsonc`:**
-
-- `account_id` → same account id
-- `routes[0]` → `{ "pattern": "*.canopy.website/*", "zone_name": "canopy.website" }`
-  (or `*.thanarie.dev/*` for staging)
-- `vars.COOKIE_DOMAIN` / `AUTH_HOST` / `API_HOST` → that site's hosts
-
-Also point the CLI at the site (`canopy-dev --profile main` / `staging`, or
-`CANOPY_API_URL=https://api.<domain>`). Until the workers exist, skip the
-CLI bits and come back in step 10.
-
-### 5. Runtime tokens (script)
-
-Mint with **token B** (API Tokens · Write). Token A cannot create other
-tokens if it lacks that scope (and a token that _has_ Tokens·Write cannot
-always be used as a "sub-token" parent for another Tokens·Write policy):
-
-```bash
-cd infra
-set -a && source .env && set +a
-export CLOUDFLARE_API_TOKEN="$CLOUDFLARE_TOKEN_MINT_TOKEN"
-CANOPY_ACCOUNT_ID="$(tofu output -raw account_id)" \
-  ./scripts/create-api-tokens.sh | tee .env.runtime
-```
-
-Prints `CF_API_TOKEN` and `AI_GATEWAY_TOKEN` once. Keep them for secrets
-(`.env.runtime` is gitignored via `.env.*`).
-
-### 6. Apply the control-plane D1 schema
-
-The Tofu-created D1 is empty. Load the schema into the remote (production) DB —
-the `db:setup` script targets `--local`, so pass `--remote` explicitly:
-
-```bash
-cd ../apps/api
-# Use the site config that matches the account (main = canopy.website).
-pnpm exec wrangler d1 execute canopy-control --remote --file schema.sql \
-  -c wrangler.main.jsonc
-```
-
-Do **not** run `seed.local.sql` against remote — it's local dev seed data. Your
-user row doesn't exist yet either; it's auto-provisioned on first login, and
-admin is granted in step 10.
-
-### 7. Deploy the platform workers
-
-`wrangler secret put` needs the Worker script to exist, so deploy **before**
-setting secrets. Use each package's `run deploy` script — bare
-`pnpm --filter api deploy` is pnpm's own `deploy` command and will fail with
-`ERR_PNPM_INVALID_DEPLOY_TARGET`:
-
-```bash
-# from repo root — pick the site (deploy aliases deploy:main)
-pnpm --filter api run deploy:main        # or deploy:staging
-pnpm --filter gateway run deploy:main    # same site
-```
-
-Sanity check: `curl -I https://auth.canopy.website` should 302 toward
-`/auth/login` (gateway alive). `curl -I https://auth.canopy.website/auth/login`
-should 302 to OneLogin with
-`redirect_uri=https://auth.canopy.website/auth/callback`.
-
-### 8. Worker secrets (script)
-
-```bash
-cd infra
-set -a && source .env.runtime && set +a   # CF_API_TOKEN + AI_GATEWAY_TOKEN
-# OIDC_CLIENT_SECRET: same OneLogin confidential-client secret as before if
-# the app is unchanged. SESSION_SECRET: blank prompt = generate a fresh one.
-OIDC_CLIENT_SECRET="…" CF_API_TOKEN="…" AI_GATEWAY_TOKEN="…" \
-  ./scripts/bootstrap-secrets.sh main    # or: staging
-```
-
-Sets secrets on both workers; **`SESSION_SECRET` is written identically to api
-and gateway** — the script handles that. Secrets take effect immediately (no
-redeploy required).
-
-### 9. OneLogin (external, dashboard)
-
-Only if the host changed and the apps aren't already pointed at it. OneLogin
-matches redirect URIs exactly (no wildcards):
-
-- Gateway confidential client → `https://auth.canopy.website/auth/callback`
-- CLI PKCE client → unchanged (`http://127.0.0.1:8976/callback`, local)
-
-If you already configured these for `canopy.website` before the move, skip.
-
-### 10. Log in, grant yourself admin, then deploy the core projects
-
-The four core projects are ordinary platform projects that dogfood the product.
-Three of their slugs (`dashboard`, `www`, `docs`) are in `RESERVED_SLUGS`
-(`apps/api/src/routes/projects.ts`), so **only an admin can create them** —
-hence the ordering here.
-
-Point the local CLI at the new platform (and rebuild if you just changed
-defaults):
-
-```bash
-# canopy-dev = this repo's CLI (packages/cli → vp pack), not the npm `canopy`
-export CANOPY_API_URL="https://canopy-api.monks.workers.dev"
-export CANOPY_PROJECT_DOMAIN="canopy.website"
-export CANOPY_PROJECT_PROTOCOL="https"
-```
-
-1. `canopy-dev login` → browser PKCE against OneLogin; your user row is
-   auto-provisioned in the new control DB. Confirm with `canopy-dev whoami`
-   (apiUrl should be the new workers.dev host).
-2. Grant admin (the row only exists after step 1). The first admin on a fresh
-   control DB still needs SQL; after that, admins can promote others from the
-   dashboard **Users** page:
-   ```bash
-   pnpm --filter api exec wrangler d1 execute canopy-control --remote \
-     --command "UPDATE users SET is_admin = 1 WHERE email = 'you@monks.com'"
-   ```
-3. Register each project remotely, then deploy it. **`--no-scaffold` is
-   required** — `create` otherwise writes a starter template into `./<slug>/`
-   (and prompts for a template picker), which inside an existing app means a
-   junk `apps/dashboard/dashboard/` directory. `--no-scaffold` registers only,
-   no files written. The `--slug` must match the `slug` in that directory's
-   existing `canopy.json`, since that's what `deploy` publishes to:
-
-   ```bash
-   # Always pass --profile <site> for the destination (main / staging / gcp-pilot).
-   # Examples below use main (production dogfood); swap the id when targeting another site.
-   cd apps/dashboard
-   canopy-dev --profile main create "Canopy Dashboard" --slug dashboard --no-scaffold
-   canopy-dev --profile main deploy
-
-   cd ../www
-   canopy-dev --profile main create "Canopy" --slug www --no-scaffold
-   canopy-dev --profile main deploy
-
-   cd ../docs
-   canopy-dev --profile main create "Canopy Docs" --slug docs --no-scaffold
-   canopy-dev --profile main deploy
-
-   cd ../../examples/kitchen-sink
-   canopy-dev --profile main create "Kitchen Sink" --slug kitchen-sink --no-scaffold
-   canopy-dev --profile main deploy
-   ```
-
-   `canopy.json` already carries the slug, build command, dist dir, and a
-   usual `"profile": "main"` pin, so `deploy` builds them — no separate
-   `pnpm build` needed. Still pass `--profile` for the intended site so a
-   stray active profile cannot send the deploy elsewhere
-   (docs/10-cli-sites.md). On a site without worker support
-   (`capabilities.workers = false`), a deploy that keeps
-   `"worker": "worker/index.ts"` will 501 `not_supported_on_target` — strip
-   the `worker` key for an assets-only deploy there; that's still enough to
-   exercise `db` / `blob` / `ai.chat` through the platform data plane.
-
-### 11. AI Gateway keys + spend rule (script)
-
-```bash
-cd infra   # adjust if you're coming from examples/kitchen-sink
-set -a && source .env.runtime && set +a
-# CF_API_TOKEN has AI Gateway Read (use that for the logs check);
-# the script's prompt is about dashboard BYOK / spend-limit clicks.
-export CLOUDFLARE_API_TOKEN="$CF_API_TOKEN"
-CANOPY_ACCOUNT_ID="$(tofu output -raw account_id)" DEFAULT_BUDGET_USD=25 \
-  ./scripts/bootstrap-ai-gateway.sh
-```
-
-Verifies the gateway, then walks you through adding BYOK provider keys and the
-per-project spend-limit rule (both dashboard steps — see the script's notes).
-`AI_GATEWAY_TOKEN` alone is **Run**-scoped and cannot read logs; use
-`CF_API_TOKEN` (D1 Write + AI Gateway Read) for the read-back check.
-
-The hard daily budget cap is enforced by canopy-api's own `ai_spend` ledger on
-every request; the AI Gateway spend rule is belt-and-braces on top.
-
-### 12. Smoke test
-
-Anonymous visits to project hosts **302 to OneLogin** by design (monk SSO;
-same as the old `thanarie.dev` setup). Client-password mode is the only
-anonymous path. Smoke through a real browser session:
-
-1. `https://dashboard.canopy.website` → OneLogin login works; the project list
-   shows all four under **Core projects**.
-2. `https://www.canopy.website` → marketing site + `/changelog` render (after
-   SSO).
-3. `https://kitchen-sink.canopy.website` → exercise `db`, `blob`, and `ai.chat`
-   (proves per-project D1 provisioning, R2 prefixes, and the AI Gateway path —
-   BYOK must be set first for `ai.chat`).
-4. Confirm kitchen-sink's LLM spend appears in its dashboard usage view (the
-   view reads the platform's own `ai_spend` ledger, written at call time — the
-   same rows the daily budget cap enforces).
-5. `canopy-dev create scratch-…` then `canopy-dev delete <slug> --yes` →
-   confirms provisioning and full teardown.
-
-CLI sanity (no browser): `curl -sS https://auth.canopy.website/auth/login`
-should 302 to OneLogin; `curl -sS https://canopy-api.<subdomain>.workers.dev/control/v1/docs`
-should 401 `unauthenticated` (API up, auth required).
-
----
-
-## Teardown of the old account
-
-Once the new account is verified and you no longer need the old projects'
-data, delete resources in the old account (or leave it to lapse). Site
-bindings live in `wrangler.main.jsonc` / `wrangler.staging.jsonc`; local
-`wrangler.jsonc` is not a deploy target.