@odla-ai/cli 0.4.0 → 0.6.0

package/skills/odla-migrate/references/phase-3-auth.md

@@ -3,31 +3,48 @@
 Goal: Clerk sign-in on the site, the worker verifying session JWTs
 itself, at least one route gated. Dev only.
 
-Human obligation: create a Clerk application (dev instance) and paste
-its PUBLISHABLE key (`pk_test_…`) when asked. Tell them explicitly:
-publishable keys are public by design — this is the only key-like value
-they will ever paste into chat. Never ask for `sk_…` or `whsec_…`.
+Human obligation: **run `npx clerk auth login`** — a one-time browser
+OAuth into their Clerk account. This is the ONE interactive step and the
+whole reason to surface it early: the agent cannot authenticate as the
+user. After login, credentials are stored locally and the agent drives
+the rest (`apps create`, `link`, `env pull`, `config patch`) from Bash.
+There is no `pk_` to hand-paste — the CLI pulls it. (Never handle `sk_…`
+or `whsec_…` in client mode; `env pull` writes an `sk_` into a gitignored
+`.env.local` that stays local and unused.)
+
+Use the Clerk CLI (`npx clerk`) to create the instance rather than the
+dashboard. Do NOT assume an app exists: a stale publishable key can linger
+on the registry's app_auth record from a deleted/recreated app —
+`clerk apps list` is the source of truth for what's real.
 
 ## Steps
 
-1. Add to `odla.config.mjs`:
-   `auth: { clerk: { dev: "<pk_test_…>" } }`
-   (Inline is fine — the value is public. The `"$ENV_VAR"` indirection
-   the config template shows is optional.)
-2. `npx odla-ai provision` (idempotent) — calls setAuth for the dev env;
-   the issuer is derived from the key. The worker fetches auth config
-   from the registry's public-config at runtime, so a key rotation
-   propagates without a redeploy.
-3. `npm i jose`. In the worker (lift the exact pattern from
-   `examples/demo-app/src/worker.ts`):
+1. Provision the Clerk app with the CLI (after the human's `clerk auth
+   login`):
+   - `npx clerk apps list` — confirm what exists (catch stale/leftover
+     instances); create if absent.
+   - `npx clerk apps create "<App Name>"` — creates the app + dev
+     instance (`pk_test_…`).
+   - `npx clerk link` (or `--app <id>`), then `npx clerk env pull` — pull
+     the dev `pk_test_…` into `.env.local`. Take only the publishable key.
+2. Add the key to `odla.config.mjs`: `auth: { clerk: { dev: "<pk_test_…>" } }`
+   (inline is fine — it's public), then `npx @odla-ai/cli provision`
+   (idempotent) — calls setAuth for the dev env; the issuer is derived
+   from the key. **provision OVERWRITES any stale key** on public-config.
+   The worker fetches auth config from public-config at runtime, so a key
+   change propagates without a redeploy.
+3. `npm i jose`. In the worker, use this documented verification pattern:
    - fetch `<ODLA_PLATFORM>/registry/apps/<ODLA_APP_ID>/public-config?env=<ODLA_ENV>`,
      cache ~5 min per isolate
    - `createRemoteJWKSet(new URL(issuer + "/.well-known/jwks.json"))`,
      cached per issuer
    - `jwtVerify(token, jwks, { issuer })` on the `Authorization: Bearer`
      header; pass `{ sub, email }` to routes
-   - ensure the Clerk session token includes an email claim (Clerk
-     dashboard → session token customization) if routes need email
+   - ensure the Clerk session token includes an email claim if routes
+     need email: try `npx clerk config schema --keys session` then
+     `npx clerk config patch --json '{…}'`; else set it in the dashboard
+     (Sessions → customize session token) to
+     `{"email": "{{user.primary_email_address}}"}`
 4. Gate at least one `/api/*` route on a verified user; return 401
    otherwise.
 5. Add Clerk sign-in to the site pages. For a React/Preact site, use
@@ -36,17 +53,86 @@ they will ever paste into chat. Never ask for `sk_…` or `whsec_…`.
    clerk-js (no clerk-react), loads clerk-js as a lazy chunk, and themes
    to odla-ui via `clerkAppearanceFromTokens()`. Otherwise drive ClerkJS
    directly with the publishable key from public-config, or use Clerk's
-   hosted pages.
+   hosted pages. For a **vanilla/static site**, load Clerk's browser bundle
+   with the key on the tag — it hot-loads `window.Clerk` as a ready
+   *instance* (not a constructor): `<script src="https://<frontend-api>/
+   npm/@clerk/clerk-js@5/dist/clerk.browser.js" data-clerk-publishable-key=
+   "<pk>">` then `await window.Clerk.load()`. The `<frontend-api>` is the
+   host encoded in the pk. Then mount `Clerk.mountSignIn(el, { appearance })`
+   and attach `Clerk.session.getToken()` as the `Bearer` on `/api/*` calls.
 6. Deploy dev (`npm run deploy:app:dev`).
 
-Auth mode "full" (webhook-synced $users via svix, whsec in the tenant
-vault) is a later upgrade — not needed to ship login.
+Clerk is the **source of truth** for identity; odla-db keeps a mirror in
+`$users`. Shipping login (this phase) is enough to gate routes. Mirroring
+users into `$users` (svix webhook, "full" mode) is a **separate, optional
+next step — see phase-3b-user-sync.md** — not needed to ship login.
+
+## Shape the instance with the Clerk CLI (optional, for invite-only sites)
+
+The same `clerk` CLI configures the instance as code (per app, from any
+directory with `--app <id>`) — no dashboard clicking. For a members-only
+site you typically want email-only + no social SSO + an allowlist:
+
+- Inspect: `npx clerk config schema --keys session auth_email auth_password
+  auth_access_control` ; pull current with `npx clerk config pull`.
+- Apply a partial patch (dry-run first with `--dry-run`):
+  `npx clerk config patch --json '{"auth_password":{"enabled":false},
+  "connection_oauth_google":{"enabled":false},
+  "auth_access_control":{"allowlist_enabled":true,
+  "allowlist_blocklist_enforced_on_sign_in":true}}'`
+  (email-code sign-in stays on via `auth_email`). A committed
+  `clerk-config.json` + `clerk config patch --file` keeps it reproducible and
+  reusable across the dev and prod instances.
+- Invite-gate sign-ups by adding allowed emails to the allowlist:
+  `npx clerk api /allowlist_identifiers -X POST -d
+  '{"identifier":"<email>","notify":false}'` (per app with `--app <id>`).
+- The email session-token claim (Phase 3 step 3) is just another patch:
+  `clerk config patch --json '{"session":{"claims":{"email":
+  "{{user.primary_email_address}}"}}}'`.
+
+## Match the site's existing look and feel (do NOT invent a style)
+
+Every visible element you add — the sign-in gate, any nav/top bar, an
+admin console, cross-links — MUST look like it was always part of the
+site. A bolted-on or default-styled element is a defect even if it
+works. Before writing any markup:
+
+1. **Read the site's design system first.** Find its stylesheet(s) and
+   shared components (header/nav/footer). Note the exact tokens: fonts
+   (heading vs body), color palette, logo/wordmark markup, spacing,
+   button styles, radii. Reuse them verbatim — same classes, or copy the
+   exact values. Never approximate; "close enough" reads as wrong (a
+   32px logo next to the real 36px one, the accent colour off by a shade,
+   a heading in the body font — all instantly visible).
+2. **Reuse existing components.** If the site has a `<site-header>`/nav
+   component, extend or replicate it exactly rather than inventing a new
+   bar. New surfaces (admin, member area) should share ONE bar with the
+   rest of the site — identical logo size, wordmark size, padding, max
+   width.
+3. **Theme third-party widgets to the brand.** A vendor's default chrome
+   (Clerk's card + purple user-button avatar, Stripe's default inputs)
+   is off-brand by definition. Use its theming API — Clerk `appearance`:
+   strip the card to blend into a host card, set `colorPrimary`/fonts/
+   radii to the site tokens, hide redundant headers, `overflow:visible`
+   so it can't clip its own labels. Don't ship a raw vendor widget, and
+   don't show an off-brand account avatar when a plain "Sign out" fits.
+4. **Bump the cache-bust version** (`?v=`) on any shared asset you change
+   (stylesheet, header script), on every page that embeds it — otherwise
+   cached visitors keep the old copy and "your fix didn't deploy".
+5. **You cannot see the render — so never claim a visual match.** State
+   what you changed and ask the human to confirm; expect a round or two
+   of "the logo's too small / it clips / that colour is wrong" and treat
+   that as the normal loop, not something to argue with.
 
 ## Verification checklist
 
 - [ ] Unauthenticated curl to the gated route → 401
 - [ ] Signed-in browser session reaches the gated route
-- [ ] `npx odla-ai smoke --env dev` still passes
+- [ ] `npx @odla-ai/cli smoke --env dev` still passes
+- [ ] Every added UI element matches the site's existing look (fonts,
+      palette, logo, spacing) — human-confirmed, not self-asserted
+- [ ] Post-login redirect lands on a real URL (mind trailing-slash /
+      directory-index routing), not a 404
 - [ ] MIGRATION.md updated (gated routes, Clerk instance name)
 
 Rollback: remove the auth layer / ungate the route; nothing outside dev

package/skills/odla-migrate/references/phase-3b-user-sync.md

@@ -0,0 +1,57 @@
+# Phase 3b — Mirror Clerk users into odla-db (`$users`)
+
+Optional, and staged: ship login first (Phase 3), then turn this on. **Clerk is
+the source of truth for identity.** odla-db keeps a *mirror* of your users in the
+reserved `$users` namespace so rules and app data can reference them. You do not
+build the mirror yourself — the platform does it. This phase enables it.
+
+Goal: signed-in users appear as rows in the app's `$users`, kept in sync with
+Clerk (create/update/delete). Dev first, then prod.
+
+Human obligation: create a Clerk webhook endpoint and paste its signing secret
+(`whsec_…`) into Studio (write-only). That plus the Phase-3 `clerk auth login`
+are the only human Clerk steps.
+
+## How `$users` fills — two paths
+
+1. **Browser talks to odla-db directly with its Clerk session token.** The
+   platform verifies the JWT and auto-creates a basic `$users` row (`id` = the
+   Clerk `sub`). No setup. This only happens for *user-token* access.
+2. **The Clerk webhook (full profile).** Clerk pushes `user.created/updated/
+   deleted` to odla-db, which upserts the full profile (`email`, `name`,
+   `imageUrl`) and tombstones on delete. Works for **any** app.
+
+**If your app is worker-mediated** (the worker holds the app's admin/scoped key
+and calls odla-db itself — the Phase-2/3 pattern), that scoped credential is
+**not projected into `$users`**. So the webhook (path 2) is how you get `$users`.
+Enable it here.
+
+## Enable the webhook
+
+1. **Create the endpoint in Clerk** (per instance — dev instance → dev tenant;
+   prod instance → prod tenant). URL:
+   `https://db.odla.ai/webhooks/clerk/<ODLA_TENANT>`
+   (the odla tenant, e.g. `<appId>--dev` in dev, `<appId>` in prod). Subscribe to
+   `user.created`, `user.updated`, `user.deleted`. Clerk returns a signing secret
+   `whsec_…`. Prefer the Clerk CLI if it exposes webhook creation; otherwise the
+   Clerk dashboard → Webhooks → Add Endpoint. (`whsec_` is a real secret — never
+   print/commit it; it is the only thing you paste, and it goes into the vault,
+   not chat.)
+2. **Store the secret in the tenant vault** as `clerk_webhook_secret`, write-only,
+   via Studio's secret UI (`odla.ai/studio` → the app → the env → secrets). The server
+   decrypts it only to verify inbound events (svix HMAC over `id.timestamp.body`).
+3. **(Optional) `sk_…`** — the webhook payload already carries email/name/image,
+   so the basic mirror needs only `whsec_`. Add the Clerk backend key `sk_…` to
+   the vault only if you later need to resolve data the webhook doesn't send.
+
+## Verify
+
+- Sign in (or edit the user in Clerk) → a `$users` row appears (check Studio, or
+  a worker route that runs `db.query({ $users: {} })`).
+- Delete the Clerk user → the row is tombstoned, not hard-deleted.
+- Rules can now gate on it: `$users` is platform-managed (a browser may view its
+  own row but cannot write `$users`). Your app's own tables (members, roles,
+  status) layer on top, keyed by the Clerk `sub` or `email`.
+
+Done when: a real sign-in produces a `$users` row in the dev tenant. Repeat the
+endpoint + secret for the prod instance/tenant at Phase 5.

package/skills/odla-migrate/references/phase-4-ai.md

@@ -15,22 +15,23 @@ own terminal so you never see the value.
    via this session):
 
        export <PROVIDER>_API_KEY=...    # their key
-       npx odla-ai provision
+       npx @odla-ai/cli provision
 
    provision stores the key in the tenant vault and sets provider/model
    in the registry. Their shell forgets it when closed; wrangler and git
-   never see it. If they use `! npx odla-ai provision` in-session, the
+   never see it. If they use `! npx @odla-ai/cli provision` in-session, the
    export must still happen in a terminal you don't read.
 3. `npm i @odla-ai/ai`. In the worker, use `initFromPlatform` — it reads
    provider/model from public-config (cached ~60s) and resolves the key
    from the vault at call time using only the app's db key. Provider and
-   model are switchable in Studio with no redeploy. Reference:
-   `examples/kg-demo` in the odla-ai monorepo.
+   model are switchable in Studio with no redeploy. Read
+   `node_modules/@odla-ai/ai/llms.txt` and the public platform manual for the
+   complete vault + public-config contract.
 4. Add one `/api/*` route that round-trips the model; deploy dev.
 
 ## Verification checklist
 
-- [ ] `npx odla-ai smoke --env dev` passes
+- [ ] `npx @odla-ai/cli smoke --env dev` passes
 - [ ] The AI route returns a model response on the deployed dev worker
 - [ ] `wrangler.jsonc` `vars` contain NO provider key; `git grep` for the
       key's prefix finds nothing

package/skills/odla-migrate/references/phase-5-cutover.md

@@ -5,25 +5,39 @@ worker, DNS cut over — with GitHub Pages kept live as the rollback for
 at least 72 hours.
 
 Human obligations: add the domain to Cloudflare; click through the DNS
-changes (you supply exact values); create the prod Clerk instance if
-the app uses login; final go/no-go at each step below.
+changes (you supply exact values); if using login, **activate production
+on the existing Clerk app** (`npx clerk deploy`) — the same app's prod
+instance, never a second Clerk app per env; final go/no-go at each step.
 
 ## Steps
 
 1. Update `odla.config.mjs`: add `"prod"` to `envs`; add
-   `auth.clerk.prod` (`pk_live_…` from the prod Clerk instance) if using
-   login; add `links: { prod: "https://<domain>" }` (the CNAME domain
-   captured in Phase 0). If using AI, the human re-runs the Phase 4
-   export + provision so the PROD tenant's vault gets the key.
-2. `npx odla-ai provision --dry-run`, show the human, then
-   `npx odla-ai provision` — provisions the prod tenant (`<appId>`),
-   idempotent for everything already done in dev.
-3. Push the prod db key: `npx odla-ai secrets push --env prod --yes`
-   (targets the top-level wrangler env; `--yes` is the explicit prod
-   consent — see references/secrets-map.md).
-4. Build, then `npx wrangler deploy` (first prod deploy). Verify
+   `auth.clerk.prod` (`pk_live_…` from the app's **activated production
+   instance** — one Clerk app, two instances, see below) if using login;
+   add `links: { prod: "https://<domain>" }` (the CNAME domain captured in
+   Phase 0). If using AI, the human re-runs the Phase 4 export + provision
+   so the PROD tenant's vault gets the key.
+2. `npx @odla-ai/cli provision --dry-run`, show the human, then
+   `npx @odla-ai/cli provision --yes --push-secrets` — provisions the prod
+   tenant (`<appId>`) and transfers its configured db/o11y secrets through
+   Wrangler stdin. `--yes` is the explicit production consent; use standalone
+   `secrets push --env prod --yes` only to retry the transfer (see
+   references/secrets-map.md).
+3. Build, then run the passive pre-cutover security gate:
+   first require `npm view @odla-ai/security@0.2.0 version` to succeed. An
+   exact-version `E404` means the release is unavailable and blocks cutover;
+   it is not a clean scan and does not prove the package name is absent. Run
+   `npm i -D --save-exact @odla-ai/security@0.2.0` followed by
+   `npx odla-security scan . --profile odla --out .odla/security/pre-cutover --fail-on high --fail-on-candidates critical`.
+   Review `REPORT.md`; a candidate is a lead, not confirmation, and a baseline
+   requires a concrete reason, owner, and expiry.
+   If the human explicitly approves redacted tracked-source disclosure, follow
+   with `npx @odla-ai/cli security run . --env prod --ack-redacted-source`.
+   The platform owns provider credentials and attributes the bounded,
+   independent-model run to this app; never request provider keys locally.
+4. `npx wrangler deploy` (first prod deploy). Verify
    `/api/health` and the parity paths on the prod workers.dev URL.
-5. `npx odla-ai smoke --env prod`.
+5. `npx @odla-ai/cli smoke --env prod`.
 6. Human: add the domain to Cloudflare, then attach it to the prod
    worker (Workers & Pages → the worker → Domains & Routes). Supply
    them the exact hostname values to enter.
@@ -37,9 +51,26 @@ the app uses login; final go/no-go at each step below.
     confirmation: disable GitHub Pages in the repo settings. KEEP the
     repo — it is still the source of the site.
 
+## Clerk in prod — one app, two instances
+
+A Clerk **application** has a Development instance (used in dev) and a
+Production instance. Production (`pk_live_…`) is **domain-coupled**:
+`npx clerk deploy` walks the human through the production domain + DNS —
+the same domain you're cutting over here — so activating it fits naturally
+at Phase 5. **Do NOT create a second Clerk app per environment** — activate
+production on the *existing* app. Then:
+
+- Re-apply the instance config to the prod instance:
+  `npx clerk config patch --instance prod --file clerk-config.json`, and
+  re-add allowlist identifiers with `--app <id>` (see phase-3-auth.md).
+- If mirroring users, add the `$users` webhook for the prod instance/tenant
+  (`https://db.odla.ai/webhooks/clerk/<appId>` + `whsec_` in the prod
+  tenant vault) — see phase-3b-user-sync.md.
+
 ## Verification checklist
 
-- [ ] `npx odla-ai smoke --env prod` passes
+- [ ] `npx @odla-ai/cli smoke --env prod` passes
+- [ ] Passive `odla-security` pre-cutover report reviewed; critical lead gate passes
 - [ ] Public domain serves from the worker (check the `x-odla-worker`
       header) on every parity path
 - [ ] Auth and AI routes verified from the public domain
@@ -49,5 +80,5 @@ Rollback: point DNS back at GitHub Pages (minutes). Nothing on the
 odla/Cloudflare side needs to be torn down to roll back.
 
 Done when: the 72-hour confirmation is done and MIGRATION.md is closed
-out. Congratulate the human — and mention Studio (https://odla.ai) as
+out. Congratulate the human — and mention Studio (https://odla.ai/studio) as
 where they watch their app from now on.

package/skills/odla-migrate/references/secrets-map.md

@@ -8,20 +8,31 @@
 | Clerk webhook secret (`whsec_…`) | tenant vault (`clerk_webhook_secret`) | only for auth mode "full"; entered in Studio, never wrangler, never chat |
 | Clerk secret key (`sk_test_`/`sk_live_`) | not used in this journey | never ask for it |
 | LLM provider key | tenant vault | env var in the HUMAN's shell for one provision run; never wrangler vars, never git, never chat |
-| `odla_sk_…` (tenant db key) | wrangler secret `ODLA_API_KEY` + `.odla/credentials.local.json` (0600) + `.dev.vars` | the only wrangler secret in this journey; move it only with the pipeline below |
+| `odla_sk_…` (tenant db key) | wrangler secret `ODLA_API_KEY` + `.odla/credentials.local.json` (0600) + `.dev.vars` | present when db is enabled; move it only with the pipeline below |
 | `odla_dev_…` (developer token) | `.odla/dev-token.json` (0600) | ~24h lifetime, provision-time only; never deployed |
-| `o11y_…` (o11y ingest token) | wrangler secret `ODLA_O11Y_TOKEN` + `.odla/credentials.local.json` (0600) + `.dev.vars` | only if the app enables o11y; provision auto-mints it (dashboard can too); moved with the pipeline below alongside the db key; never a var, never chat |
-| `ODLA_ENDPOINT` / `ODLA_TENANT` / `ODLA_PLATFORM` / `ODLA_APP_ID` / `ODLA_ENV` / `ODLA_O11Y_ENDPOINT` / `ODLA_O11Y_SERVICE` | wrangler `vars` | not secrets; keep them set in every env block |
+| scoped `odla_dev_…` (admin-approved platform action) | `.odla/admin-token.local.json` (0600) | exact visible scope, ~15m lifetime, never ambient admin or deployed |
+| `o11y_…` (o11y ingest token) | wrangler secret `ODLA_O11Y_TOKEN` + `.odla/credentials.local.json` (0600) + `.dev.vars` | only if the app enables o11y; provision issues/reuses it and moves it alongside the db key; never a var, never chat |
+| `ODLA_ENDPOINT` / `ODLA_TENANT` / `ODLA_PLATFORM` / `ODLA_APP_ID` / `ODLA_ENV` | wrangler `vars` | not secrets; keep them set in every env block |
+| `ODLA_O11Y_ENDPOINT` / `ODLA_O11Y_SERVICE` | `.dev.vars` from provision or optional wrangler `vars` overrides | not secrets; public ingest defaults to `https://o11y.odla.ai` and `ODLA_APP_ID` when the token is present |
+
+System AI provider credentials are platform-owned and never enter a customer
+migration. `odla-ai security run` receives only app/run-bound role grants; do
+not ask the human for Anthropic/OpenAI/Google keys for hosted security.
 
 ## Moving the db key (+ o11y token) into the Worker (never through the transcript)
 
-Use the CLI — it reads the 0600 credentials file and pipes each value to
-wrangler over stdin, with preflights (`wrangler whoami`, config present)
-and redacted output. It pushes `ODLA_API_KEY` and, when the app enabled o11y,
-`ODLA_O11Y_TOKEN` too:
+Normal provisioning owns the complete sequence — issuance, mode-0600 local
+storage, `.dev.vars`, and redacted Wrangler stdin transfer. It pushes
+`ODLA_API_KEY` and, when the app enabled o11y, `ODLA_O11Y_TOKEN` too:
+
+    npx @odla-ai/cli provision --write-dev-vars --push-secrets
+    npx @odla-ai/cli provision --yes --push-secrets  # Phase 5; includes prod consent
+
+Use the narrower command only to retry one environment's already-saved
+credential transfer:
 
-    npx odla-ai secrets push --env dev
-    npx odla-ai secrets push --env prod --yes   # Phase 5; --yes is the prod consent
+    npx @odla-ai/cli secrets push --env dev
+    npx @odla-ai/cli secrets push --env prod --yes
 
 Manual fallback (identical mechanics, if the CLI is unavailable):
 
@@ -41,5 +52,9 @@ wrangler env is prod.)
   but verify.
 - If a secret value ever does land in the conversation or a committed
   file: tell the human immediately, treat it as burned, and rotate it
-  (`provision --rotate-keys` for db keys — with their explicit
-  approval; provider dashboard for LLM/Clerk values).
+  (`provision --rotate-keys` for the broad credential set, or
+  `provision --rotate-o11y-token --push-secrets` for o11y only — with explicit
+  human approval; provider dashboard for LLM/Clerk values).
+- Studio's o11y token control is manual recovery only. It invalidates the live
+  token immediately and does not update `.odla/credentials.local.json`; prefer
+  the CLI rotation above so the saved and deployed values move together.

package/skills/odla-migrate/references/troubleshooting.md

@@ -22,14 +22,14 @@ Cause: tenant confusion — the code ran against `<appId>` (prod) instead
 of `<appId>--dev`, or vice versa. Typical trigger: `wrangler deploy`
 without `--env dev`, or `wrangler dev` picking up top-level vars.
 Fix: check `ODLA_TENANT`/`ODLA_ENV` in the relevant `wrangler.jsonc`
-env block and which deploy command ran. `npx odla-ai smoke --env dev`
+env block and which deploy command ran. `npx @odla-ai/cli smoke --env dev`
 prints what it verified against.
 
 ## Provision fails with an auth/token error
 
 Cause: the `odla_dev_…` token expired (~24h) or the handshake was never
 approved.
-Fix: re-run `npx odla-ai provision` — it starts a fresh handshake; the
+Fix: re-run `npx @odla-ai/cli provision` — it starts a fresh handshake; the
 human approves the new code at the printed URL. Use `--no-open` in
 non-interactive shells if the browser launch misbehaves.
 
@@ -37,16 +37,27 @@ non-interactive shells if the browser launch misbehaves.
 
 Cause: `.odla/credentials.local.json` absent, for a different app id,
 or lacking a db key for the requested env.
-Fix: run `npx odla-ai provision` for the configured envs first; confirm
+Fix: run `npx @odla-ai/cli provision` for the configured envs first; confirm
 `envs` in odla.config.mjs includes the env you're smoking.
 
 ## `smoke` fails: schema mismatch
 
 Cause: local `src/odla/schema.mjs` changed since the last push.
-Fix: re-run `npx odla-ai provision` (schema re-push is the migration
+Fix: re-run `npx @odla-ai/cli provision` (schema re-push is the migration
 mechanism; additive changes are safe on live tenants), then re-run
 smoke.
 
+## Provision reports an existing o11y token but no local credential
+
+Cause: the shown-once token was issued elsewhere, or the local credentials file
+was lost. Plain provision refuses to invalidate a deployed Worker just to
+recover plaintext.
+Fix: after explicit human approval, run
+`npx @odla-ai/cli provision --rotate-o11y-token --push-secrets` (add `--yes`
+when production is in the plan). This persists the replacement before moving it
+to the Worker. Do not use the Studio button for routine recovery; it cannot sync
+the repository's credential file.
+
 ## Clerk-verified requests return no email
 
 Cause: the Clerk session token doesn't include an email claim.
@@ -55,15 +66,18 @@ claim, then re-test. The worker only forwards what the JWT carries.
 
 ## Re-running provision — is it safe?
 
-Yes: re-runs are idempotent (existing app registration and db keys are
-reused; schema/rules re-push). The only destructive flag is
-`--rotate-keys`, which mints new db keys — deployed workers keep
-working only after you push the new key to them. Use it only on
-explicit human request (e.g. a burned key).
+Yes: plain re-runs are idempotent (existing app registration and configured db
+and o11y credentials are reused; schema/rules re-push). Destructive rotation is
+always explicit: `--rotate-o11y-token` replaces only o11y, while
+`--rotate-keys` is the broader credential rotation. Use either only on explicit
+human request, and pair rotation with `--push-secrets` so the Worker receives
+the replacement in the same run. This is not atomic: if the final Wrangler
+transfer fails, the new value is already in the private credentials file. Run
+the printed `secrets push --env ...` retry and do not rotate again.
 
 ## Something not covered here
 
-Check, in order: `npx odla-ai doctor` output; the field notes in
-https://odla.ai/llms.txt; the relevant
-`node_modules/@odla-ai/*/llms.txt`. Do not improvise around a safety
-rule to unblock yourself — surface the blocker to the human instead.
+Check, in order: `npx @odla-ai/cli doctor` output; this skill's current phase
+and troubleshooting references; then the relevant installed
+`node_modules/@odla-ai/*/llms.txt`. Do not improvise around a safety rule to
+unblock yourself — surface the blocker to the human instead.

package/skills/odla-o11y-debug/SKILL.md

@@ -0,0 +1,93 @@
+---
+name: odla-o11y-debug
+description: >
+  Triage a production issue in an app instrumented with @odla-ai/o11y — find why
+  it's erroring, why latency or cost is spiking, and what's unusual about the bad
+  requests — by reading the odla-o11y API. Use when a deployed odla app is
+  misbehaving and you need to dig into its telemetry.
+runbookOrder: []
+---
+
+# odla-o11y-debug
+
+You are debugging a **live odla app** through its observability data. The app is
+instrumented with `@odla-ai/o11y`; its traces, errors, metrics, and LLM cost land
+in the `odla-o11y` collector. This skill is the read side: how to pull the data
+and reason from a symptom to a cause.
+
+The core move is **cutting high-dimensional data to isolate the bad slice** — not
+"is p95 high" but "*which* route / status / model / user is dragging it, and what
+do those requests share."
+
+## Access
+
+All reads go through the platform shell, operator-authed, so the collector's token
+never leaves the server:
+
+```
+GET https://odla.ai/o11y/<appId>/<endpoint>?env=prod
+Authorization: Bearer <developer-token>
+```
+
+Get `<developer-token>` from the device handshake (`POST https://odla.ai/handshake`
+→ approve in the Studio → `POST /handshake/poll`). `env` is `prod` (default) or
+`dev`. If a read returns `{"error":"ae_not_configured"}` (501), the app's metrics
+reads aren't turned on yet — errors/traces still work; say so and fall back to them.
+
+## Endpoints
+
+| Endpoint | Answers |
+| --- | --- |
+| `GET /o11y/<app>/events` | recent activity feed (traces + errors), newest first |
+| `GET /o11y/<app>/errors` | deduped error groups (type, code, route, count, last seen) |
+| `GET /o11y/<app>/errors/<fingerprint>` | **one group + its message + STACK + span waterfall** |
+| `GET /o11y/<app>/traces/<traceId>` | the merged span tree for a trace |
+| `GET /o11y/<app>/metrics/red?minutes=60` | request/error/latency by route |
+| `GET /o11y/<app>/metrics/timeseries?minutes=60` | RED over time (spot the spike) |
+| `GET /o11y/<app>/explore?dimension=&metric=` | **RED grouped by ANY dimension** |
+| `GET /o11y/<app>/llm/cost?minutes=1440` | spend by provider/model |
+| `GET /o11y/<app>/llm/cost/timeseries` | spend over time |
+| `GET /o11y/<app>/alerts` | configured triggers + firing history |
+
+`explore` dimensions: `route`, `status`, `name`, `kind`, `http_status`,
+`http_method`, `env`. `metric` (what to rank by): `p95_ms`, `avg_ms`, `requests`,
+`errors`.
+
+## Triage loops
+
+**An error is reported.** `GET /errors` → find the group by type/route/count →
+`GET /errors/<fingerprint>`. The response `bundle` carries the **message and full
+stack** (recovered from the logs signal — this is the whole point; older builds
+dropped it). Read the stack, then read the `spans` waterfall to see which
+downstream call (db, LLM, fetch) failed or was slow. Name the failing line.
+
+**Latency or error rate is spiking ("why is p95 up?").** This is the exploration
+loop:
+1. `GET /metrics/timeseries?minutes=<window>` — confirm the spike and its window.
+2. `GET /explore?dimension=route&metric=p95_ms` — which route owns the p95? Then
+   pivot the dimension: `status`, `http_status`, `http_method`, `env` — keep
+   slicing until one value dominates the metric.
+3. Pull a couple of the offending traces (`/events` filtered to that route, then
+   `/traces/<id>`) and read their spans + attributes. State **what the bad
+   requests share** (a route, a status code, a slow db/LLM span, a deploy env).
+
+**Cost is climbing.** `GET /llm/cost/timeseries` (when) + `GET /llm/cost` (which
+model). A single expensive model or a runaway call count is usually the story.
+
+**"Is anything already watching this?"** `GET /alerts` — configured triggers and
+their firing history (with per-channel delivery). If a spike had no trigger,
+suggest one (metric + threshold + window).
+
+## Reasoning discipline
+
+- **Lead with the stack.** For any error, quote `bundle.message` and the top of
+  `bundle.stack` before theorizing — it usually names the cause outright.
+- **Isolate before explaining.** Don't explain an aggregate ("p95 is 800ms");
+  slice with `explore` until you have the specific cohort, then explain that.
+- **Attribute, don't guess.** "These slow requests are all `http_status=503` on
+  `POST /pay`" beats "the payment service might be slow."
+- **Weighted counts.** Metric aggregates are sampling-weighted (Analytics Engine
+  adaptively samples under load) — treat counts as estimates, ratios as sound.
+- **Say what you couldn't see.** Arbitrary custom attributes live in the trace
+  payload, not the group-by dimensions; if the cohort is defined by a custom
+  field, note that you inferred it from individual traces, not an aggregate.