@mediamonks/canopy-cli 0.4.5 → 0.5.1

package/dist/docs/infrastructure.md

@@ -0,0 +1,397 @@
+# Infrastructure as code & standing up a Cloudflare account
+
+Canopy's Cloudflare footprint is small on purpose: almost everything
+per-project is provisioned at runtime by canopy-api (`apps/api/src/cf.ts`,
+reached through the Cloudflare `PlatformAdapter` in
+`apps/api/src/platform/cloudflare.ts`), so the only things worth managing
+declaratively are a handful of platform-wide resources. Those live in
+[`infra/`](../../infra) as OpenTofu; the rest is bootstrap scripts and this
+runbook.
+
+**Live platform (2026-08-01):** Monks Explore account
+`1c826c7b063fd9ca5e1b1bdaabd05feb`, zone `canopy.website`. The steps below
+are the verified path used for that move (personal account
+`9513cbfa5b182546f737501b10e1ed85` / `thanarie.dev` → company). Re-run them
+to stand the platform up in another fresh account, or to rebuild after a
+wipe.
+
+Scope of a fresh stand-up:
+
+1. the **platform workers** — `canopy-api` and `canopy-gateway` — deployed and
+   healthy (these aren't optional; nothing works without them), and
+2. the **three core projects** — `dashboard`, `www`, `kitchen-sink` — re-created
+   and deployed _on_ the platform like any other project.
+
+Everyone else's projects are **not** carried over — a new account starts with
+an empty control-plane registry (see [Non-goals](#non-goals)).
+
+See decision **D14** in [`02-decisions.md`](../02-decisions.md) for the why.
+
+## What's code vs. what's clicked
+
+**OpenTofu** (`infra/*.tf`): zone, proxied wildcard DNS record, control-plane
+D1, R2 bucket, AI Gateway shell, optional ACM cert pack.
+
+**wrangler** (`apps/{api,gateway}/wrangler.{main,staging}.jsonc`): the two
+platform Worker scripts, their bindings, and the `*.<zone>/*` route — one
+config file per Cloudflare site (same names as CLI profiles). Deployed with
+`pnpm run deploy:main` / `deploy:staging`, not Tofu. Plain `wrangler.jsonc`
+is local `wrangler dev` only.
+
+**`canopy deploy`** (`canopy.json` in each project): the core projects
+(`dashboard`, `www`, `kitchen-sink`) — ordinary platform projects that
+dogfood the product, not infrastructure.
+
+**Bootstrap scripts** (`infra/scripts/`): runtime API tokens, worker secrets,
+AI Gateway BYOK keys + spend rule.
+
+**Manual / billing (clicked once in the dashboard):** create the account &
+attach it to the org; add the domain; mint bootstrap API token(s); optionally
+enable paid Workers products. Plus OneLogin: redirect URIs for the new host
+(if they aren't already set).
+
+## Non-goals
+
+- **No project-data migration.** Per-project D1s, R2 blobs, and deploy history
+  stay in the old account. The new control-plane D1 is empty; the three core
+  projects are re-created from scratch (they hold no precious data — `www` and
+  `dashboard` are stateless, `kitchen-sink` is a demo), and everyone else's
+  projects are re-created as needed. (If that ever changes, the export/import
+  primitives already exist in `apps/api/src/cf.ts` and `canopy db snapshot|restore`.)
+- **No Tofu management of Workers/secrets/per-project resources** — see the
+  table in [`infra/README.md`](../../infra/README.md).
+- **No per-project Workers on the Cloudflare target today.** The CF
+  `PlatformAdapter` reports `capabilities.workers = false` (and
+  schedules/secrets likewise). Project backends, schedules, and project
+  secrets live on targets that support them (e.g. GCP). Kitchen-sink's
+  `worker` field in `canopy.json` must be omitted for a CF deploy — see
+  [step 10](#10-log-in-grant-yourself-admin-then-deploy-the-core-projects).
+
+---
+
+## Rollout — step by step
+
+Prereqs on your machine: `tofu` (or `terraform`), `wrangler` (via `pnpm` in
+the repo), `curl`, `jq`, `openssl`. Confirm wrangler can see the target
+account after `wrangler login`:
+
+```bash
+pnpm --filter api exec wrangler whoami
+# expect the new account listed (e.g. "Monks Explore")
+export CLOUDFLARE_ACCOUNT_ID="<account_id>"   # required when you belong to >1 account
+```
+
+### 1. Account & billing (dashboard)
+
+1. Create / choose the new account; attach it to the company org; ensure your
+   user is a member with rights to manage Workers / D1 / R2 / DNS.
+2. Add `canopy.website` to the account and repoint its nameservers at
+   Cloudflare. Wait for the zone to go **Active**. Note the **Account ID**
+   (`tofu` var `account_id`) and the **Zone ID**.
+3. **Delete any auto-created `www` record.** Cloudflare's onboarding scan often
+   adds a `www` A/CNAME. An explicit record beats the wildcard, and if it's
+   DNS-only (grey cloud) the request never reaches Workers — which would break
+   the `www` core project. The wildcard `*` alone must serve `www.canopy.website`.
+   (You can confirm via API once the bootstrap token exists — an empty zone is
+   fine.)
+4. **R2** must work (creating a probe bucket is enough to confirm). **Workers
+   Paid** and **Workers for Platforms** are _not_ required for the current CF
+   platform path (no dispatch namespaces; `capabilities.workers` is false).
+   Enable them later when you actually need paid Worker limits or Phase-2
+   per-project Workers. **No Advanced Certificate Manager** — `*.canopy.website`
+   is a single-level wildcard covered by free Universal SSL. (ACM would only
+   be needed if nesting as `*.canopy.<some-other-domain>`.)
+
+> **Apex is not covered.** `*` matches only one label, so `canopy.website`
+> itself isn't served by the gateway (by design — see
+> `apps/gateway/wrangler.main.jsonc`).
+> Decide separately whether the apex should redirect to `www.canopy.website`
+> (a Cloudflare Redirect Rule, cheap to add later); nothing in this runbook
+> depends on it.
+
+### 2. Bootstrap API token(s) (dashboard)
+
+You need credentials that can (a) run OpenTofu + wrangler ops, and (b) mint
+the least-privilege runtime tokens. In practice that is often **two**
+account-owned tokens — a single token that mixes zone DNS with **Account API
+Tokens · Write** has been flaky when edited in place.
+
+**Token A — `canopy-bootstrap` (Tofu + deploy ops)**
+
+| Resource                                                     | Permissions                                                             |
+| ------------------------------------------------------------ | ----------------------------------------------------------------------- |
+| Account                                                      | D1 Edit, Workers R2 Storage Edit, AI Gateway Edit, Workers Scripts Edit |
+| Zone `canopy.website` (nested under the account — see below) | Zone Edit, DNS Edit, Workers Routes Edit                                |
+
+**Token B — mint token (for `create-api-tokens.sh` only)**
+
+| Resource | Permissions                               |
+| -------- | ----------------------------------------- |
+| Account  | **Account API Tokens · Write** (and Read) |
+
+**Pitfalls we hit:**
+
+- The dashboard's "DNS" picks are easy to get wrong. You need zone-scoped
+  **DNS Write** / **Zone Write**, _not_ Account DNS Settings, DNS Views,
+  DNS Firewall, or Registrar Domains.
+- For **account-owned** tokens, zone resources must be **nested under the
+  account** in the policy JSON shape
+  `com.cloudflare.api.account.<account_id> → com.cloudflare.api.account.zone.<zone_id>`.
+  A top-level zone resource can look correct in the UI and still 10000-auth on
+  `/zones/:id/dns_records`.
+- `/user/tokens/verify` returns "Invalid API Token" for account-owned tokens —
+  use `GET /accounts/<account_id>/tokens/verify` instead.
+- Wrangler **OAuth** can list/create D1 and R2 and deploy Workers, but it
+  cannot manage DNS (and often cannot call AI Gateway). Don't rely on OAuth
+  alone for step 3.
+
+Store tokens outside git. Convenience pattern used in the move:
+
+```bash
+cd infra
+# gitignored via infra/.gitignore (.env / .env.*)
+cat > .env <<'EOF'
+CLOUDFLARE_API_TOKEN="<token A — canopy-bootstrap>"
+CLOUDFLARE_TOKEN_MINT_TOKEN="<token B — API Tokens Write>"
+EOF
+set -a && source .env && set +a
+```
+
+### 3. Provision static infra (OpenTofu)
+
+```bash
+cd infra
+cp terraform.tfvars.example terraform.tfvars     # set account_id + zone_name
+set -a && source .env && set +a                  # CLOUDFLARE_API_TOKEN = token A
+tofu init
+tofu import cloudflare_zone.platform <zone_id>   # adopt the domain from step 1
+tofu plan                                        # review — should be creates only
+tofu apply
+tofu output wrangler_patch_hint
+```
+
+Expect four creates: wildcard DNS `*`, D1 `canopy-control`, R2 `canopy-assets`,
+AI Gateway `canopy`. (If the AI Gateway resource fails schema validation,
+check `cache_invalidate_on_update` and friends against the pinned
+`cloudflare` provider minor — it shifted in ~> 5.22.)
+
+The R2 remote-state backend in `versions.tf` is commented out; for a solo
+stand-up, local state is fine. If you want shared state, create a
+`canopy-tfstate` bucket first, then uncomment and `tofu init -migrate-state`.
+
+### 4. Point the apps at the new resources
+
+Edit the **site** wrangler configs (not the local `wrangler.jsonc`) using
+`tofu output wrangler_patch_hint`. For production that is
+`wrangler.main.jsonc` in both apps; for the personal/staging account use
+`wrangler.staging.jsonc`. Keep both files in git — never comment-flip a
+single file to switch accounts.
+
+**`apps/api/wrangler.<site>.jsonc`:**
+
+- `account_id` / `vars.CF_ACCOUNT_ID` → account id
+- `d1_databases[0].database_id` → control-plane D1 UUID
+- `vars.AI_GATEWAY_ID` → unchanged (`canopy`) unless renamed
+- `r2_buckets[0].bucket_name` → unchanged (`canopy-assets`) unless renamed
+
+**`apps/gateway/wrangler.<site>.jsonc`:**
+
+- `account_id` → same account id
+- `routes[0]` → `{ "pattern": "*.canopy.website/*", "zone_name": "canopy.website" }`
+  (or `*.thanarie.dev/*` for staging)
+- `vars.COOKIE_DOMAIN` / `AUTH_HOST` / `API_HOST` → that site's hosts
+
+Also point the CLI at the site (`canopy-dev --profile main` / `staging`, or
+`CANOPY_API_URL=https://api.<domain>`). Until the workers exist, skip the
+CLI bits and come back in step 10.
+
+### 5. Runtime tokens (script)
+
+Mint with **token B** (API Tokens · Write). Token A cannot create other
+tokens if it lacks that scope (and a token that _has_ Tokens·Write cannot
+always be used as a "sub-token" parent for another Tokens·Write policy):
+
+```bash
+cd infra
+set -a && source .env && set +a
+export CLOUDFLARE_API_TOKEN="$CLOUDFLARE_TOKEN_MINT_TOKEN"
+CANOPY_ACCOUNT_ID="$(tofu output -raw account_id)" \
+  ./scripts/create-api-tokens.sh | tee .env.runtime
+```
+
+Prints `CF_API_TOKEN` and `AI_GATEWAY_TOKEN` once. Keep them for secrets
+(`.env.runtime` is gitignored via `.env.*`).
+
+### 6. Apply the control-plane D1 schema
+
+The Tofu-created D1 is empty. Load the schema into the remote (production) DB —
+the `db:setup` script targets `--local`, so pass `--remote` explicitly:
+
+```bash
+cd ../apps/api
+# Use the site config that matches the account (main = canopy.website).
+pnpm exec wrangler d1 execute canopy-control --remote --file schema.sql \
+  -c wrangler.main.jsonc
+```
+
+Do **not** run `seed.local.sql` against remote — it's local dev seed data. Your
+user row doesn't exist yet either; it's auto-provisioned on first login, and
+admin is granted in step 10.
+
+### 7. Deploy the platform workers
+
+`wrangler secret put` needs the Worker script to exist, so deploy **before**
+setting secrets. Use each package's `run deploy` script — bare
+`pnpm --filter api deploy` is pnpm's own `deploy` command and will fail with
+`ERR_PNPM_INVALID_DEPLOY_TARGET`:
+
+```bash
+# from repo root — pick the site (deploy aliases deploy:main)
+pnpm --filter api run deploy:main        # or deploy:staging
+pnpm --filter gateway run deploy:main    # same site
+```
+
+Sanity check: `curl -I https://auth.canopy.website` should 302 toward
+`/auth/login` (gateway alive). `curl -I https://auth.canopy.website/auth/login`
+should 302 to OneLogin with
+`redirect_uri=https://auth.canopy.website/auth/callback`.
+
+### 8. Worker secrets (script)
+
+```bash
+cd infra
+set -a && source .env.runtime && set +a   # CF_API_TOKEN + AI_GATEWAY_TOKEN
+# OIDC_CLIENT_SECRET: same OneLogin confidential-client secret as before if
+# the app is unchanged. SESSION_SECRET: blank prompt = generate a fresh one.
+OIDC_CLIENT_SECRET="…" CF_API_TOKEN="…" AI_GATEWAY_TOKEN="…" \
+  ./scripts/bootstrap-secrets.sh main    # or: staging
+```
+
+Sets secrets on both workers; **`SESSION_SECRET` is written identically to api
+and gateway** — the script handles that. Secrets take effect immediately (no
+redeploy required).
+
+### 9. OneLogin (external, dashboard)
+
+Only if the host changed and the apps aren't already pointed at it. OneLogin
+matches redirect URIs exactly (no wildcards):
+
+- Gateway confidential client → `https://auth.canopy.website/auth/callback`
+- CLI PKCE client → unchanged (`http://127.0.0.1:8976/callback`, local)
+
+If you already configured these for `canopy.website` before the move, skip.
+
+### 10. Log in, grant yourself admin, then deploy the core projects
+
+The three core projects are ordinary platform projects that dogfood the product.
+Two of their slugs (`dashboard`, `www`) are in `RESERVED_SLUGS`
+(`apps/api/src/routes/projects.ts`), so **only an admin can create them** —
+hence the ordering here.
+
+Point the local CLI at the new platform (and rebuild if you just changed
+defaults):
+
+```bash
+# canopy-dev = this repo's CLI (packages/cli → vp pack), not the npm `canopy`
+export CANOPY_API_URL="https://canopy-api.monks.workers.dev"
+export CANOPY_PROJECT_DOMAIN="canopy.website"
+export CANOPY_PROJECT_PROTOCOL="https"
+```
+
+1. `canopy-dev login` → browser PKCE against OneLogin; your user row is
+   auto-provisioned in the new control DB. Confirm with `canopy-dev whoami`
+   (apiUrl should be the new workers.dev host).
+2. Grant admin (the row only exists after step 1). The first admin on a fresh
+   control DB still needs SQL; after that, admins can promote others from the
+   dashboard **Users** page:
+   ```bash
+   pnpm --filter api exec wrangler d1 execute canopy-control --remote \
+     --command "UPDATE users SET is_admin = 1 WHERE email = 'you@monks.com'"
+   ```
+3. Register each project remotely, then deploy it. **`--no-scaffold` is
+   required** — `create` otherwise writes a starter template into `./<slug>/`
+   (and prompts for a template picker), which inside an existing app means a
+   junk `apps/dashboard/dashboard/` directory. `--no-scaffold` registers only,
+   no files written. The `--slug` must match the `slug` in that directory's
+   existing `canopy.json`, since that's what `deploy` publishes to:
+
+   ```bash
+   # Always pass --profile <site> for the destination (main / staging / gcp-pilot).
+   # Examples below use main (production dogfood); swap the id when targeting another site.
+   cd apps/dashboard
+   canopy-dev --profile main create "Canopy Dashboard" --slug dashboard --no-scaffold
+   canopy-dev --profile main deploy
+
+   cd ../www
+   canopy-dev --profile main create "Canopy" --slug www --no-scaffold
+   canopy-dev --profile main deploy
+
+   cd ../../examples/kitchen-sink
+   canopy-dev --profile main create "Kitchen Sink" --slug kitchen-sink --no-scaffold
+   # CF target: capabilities.workers = false — strip worker for this deploy
+   cp canopy.json canopy.json.bak
+   # remove the "worker" key (jq / editor), then:
+   canopy-dev --profile main deploy
+   mv canopy.json.bak canopy.json
+   ```
+
+   `canopy.json` already carries the slug, build command, dist dir, and a
+   usual `"profile": "main"` pin, so `deploy` builds them — no separate
+   `pnpm build` needed. Still pass `--profile` for the intended site so a
+   stray active profile cannot send the deploy elsewhere
+   (docs/10-cli-sites.md). A kitchen-sink deploy that keeps
+   `"worker": "worker/index.ts"` will 501 `not_supported_on_target` on
+   Cloudflare; assets-only is enough to exercise `db` / `blob` / `ai.chat`
+   through the platform data plane.
+
+### 11. AI Gateway keys + spend rule (script)
+
+```bash
+cd infra   # adjust if you're coming from examples/kitchen-sink
+set -a && source .env.runtime && set +a
+# CF_API_TOKEN has AI Gateway Read (use that for the logs check);
+# the script's prompt is about dashboard BYOK / spend-limit clicks.
+export CLOUDFLARE_API_TOKEN="$CF_API_TOKEN"
+CANOPY_ACCOUNT_ID="$(tofu output -raw account_id)" DEFAULT_BUDGET_USD=25 \
+  ./scripts/bootstrap-ai-gateway.sh
+```
+
+Verifies the gateway, then walks you through adding BYOK provider keys and the
+per-project spend-limit rule (both dashboard steps — see the script's notes).
+`AI_GATEWAY_TOKEN` alone is **Run**-scoped and cannot read logs; use
+`CF_API_TOKEN` (D1 Write + AI Gateway Read) for the read-back check.
+
+The hard daily budget cap is enforced by canopy-api's own `ai_spend` ledger on
+every request; the AI Gateway spend rule is belt-and-braces on top.
+
+### 12. Smoke test
+
+Anonymous visits to project hosts **302 to OneLogin** by design (monk SSO;
+same as the old `thanarie.dev` setup). Client-password mode is the only
+anonymous path. Smoke through a real browser session:
+
+1. `https://dashboard.canopy.website` → OneLogin login works; the project list
+   shows all three under **Core projects**.
+2. `https://www.canopy.website` → marketing site + `/changelog` render (after
+   SSO).
+3. `https://kitchen-sink.canopy.website` → exercise `db`, `blob`, and `ai.chat`
+   (proves per-project D1 provisioning, R2 prefixes, and the AI Gateway path —
+   BYOK must be set first for `ai.chat`).
+4. Confirm kitchen-sink's LLM spend appears in its dashboard usage view (the
+   view reads the platform's own `ai_spend` ledger, written at call time — the
+   same rows the daily budget cap enforces).
+5. `canopy-dev create scratch-…` then `canopy-dev delete <slug> --yes` →
+   confirms provisioning and full teardown.
+
+CLI sanity (no browser): `curl -sS https://auth.canopy.website/auth/login`
+should 302 to OneLogin; `curl -sS https://canopy-api.<subdomain>.workers.dev/control/v1/docs`
+should 401 `unauthenticated` (API up, auth required).
+
+---
+
+## Teardown of the old account
+
+Once the new account is verified and you no longer need the old projects'
+data, delete resources in the old account (or leave it to lapse). Site
+bindings live in `wrangler.main.jsonc` / `wrangler.staging.jsonc`; local
+`wrangler.jsonc` is not a deploy target.