@lunora/cli 1.0.0-alpha.48 → 1.0.0-alpha.480

package/skills/lunora-migration-helper/SKILL.md

@@ -1,194 +1,239 @@
 ---
 name: lunora-migration-helper
-description: Plans Lunora schema and data migrations with widen-migrate-narrow. Use for
-    breaking schema changes, backfills, table reshaping, online data migrations
-    (`defineMigration` + `lunora migrate up`), the `.global()` D1 SQL flow, and the
-    pre-deploy schema-drift gate.
+description: Plans and runs Lunora schema and data migrations with the widen → migrate → narrow pattern. Covers online data migrations (`defineMigration` in `lunora/migrations.ts`, `lunora migrate create | up | down | status`), what the pre-deploy schema-drift gate treats as breaking and how a migration covers it, `.global()` table DDL on D1 (`lunora migrate generate`) and Hyperdrive, moving a dataset with `lunora migrate d1-to-hyperdrive` (and back with `hyperdrive-to-d1`), and re-keying a `.shardBy()` table. Use when the user wants to make a field required, rename or drop a field or table, change a field's type, add a unique index to a populated table, backfill data, switch a table between root / `.shardBy()` / `.global()`, or when `lunora deploy` reports "schema drift gate blocked".
 ---
 
 # Lunora Migration Helper
 
-Safely change a Lunora schema and migrate data when making breaking changes.
+Change a Lunora schema that already has data in it without breaking the deploy
+or losing rows.
+
+Not needed for a greenfield schema with no data, a new optional field, a new
+table, or a new non-unique index. Those are additive and deploy as-is.
+
+## Which storage layer are you migrating?
+
+Check the table's modifier in `lunora/schema.ts` first; the tools differ.
+
+| Table                                | Where rows live                      | Structural DDL                                                 | Data backfill                          |
+| ------------------------------------ | ------------------------------------ | -------------------------------------------------------------- | -------------------------------------- |
+| default (root) or `.shardBy(key)`    | SQLite in the root / per-key ShardDO | Applied by the runtime from the schema                         | `defineMigration` + `lunora migrate`   |
+| `.global()`                          | D1 (`DB` binding)                    | Additive DDL auto-provisioned; destructive SQL applied by hand | SQL `UPDATE` via `wrangler d1 execute` |
+| `.global({ backend: "hyperdrive" })` | Postgres/MySQL via Hyperdrive        | Additive DDL auto-provisioned; no `migrate generate` support   | SQL `UPDATE` against the database      |
+
+## The pattern: widen, migrate, narrow
+
+The schema-drift gate blocks a breaking change; one that needs a backfill is
+let through once a new migration on that table covers it (the rest are listed
+below). Stage every such change across deploys:
+
+1. **Widen.** Make the schema accept both shapes: add the new field as
+   `v.optional(...)`, keep the old one. Make reads handle both, and start
+   writing the new shape on every insert/patch so rows created during the
+   migration aren't missed. Deploy.
+2. **Migrate.** Backfill existing rows with a `defineMigration`, run it with
+   `lunora migrate up`, and confirm with `lunora migrate status`. On a
+   `.global()` table, backfill with SQL instead (see below).
+3. **Narrow.** Make the field required / drop the old one, remove the
+   both-shapes code. Deploy.
+
+Prefer adding a new field to changing an existing field's type: it rolls back
+cleanly. Keep a deprecated field as `v.optional` with a `// deprecated:` comment
+until nothing reads it.
+
+## What the drift gate considers breaking
+
+The gate (run by `lunora deploy`, `prepare`, `build` and, read-only, `verify`)
+diffs the schema against the committed `lunora/.lunora-schema.json`.
+
+- **Safe:** new table, new optional field, new non-unique index or relation, a
+  required field made optional.
+- **Breaking, fixable by a migration:** dropped field or table, changed field
+  type, optional → required, new required field on an existing table, a new
+  unique index or `.unique()` column (existing duplicates would make the shard
+  fail to open).
+- **Moves data, blocks until a migration accompanies it:** changing a table's
+  shard mode (root ↔ `.shardBy()` ↔ `.global()`, or the `.global()` backend), turning
+  on `.memory()`, or changing the jurisdiction. Move the rows with export → import,
+  then declare a migration whose literal `covers` names the change
+  (`covers: { table: "users", shardMode: "shardBy:tenantId" }`, `{ table, memory: true }`
+  or `{ jurisdiction: "eu" }`). An unrelated migration clears nothing. Where a table is
+  involved its `up` must rewrite rows (give each what its new home needs, e.g. its
+  shard key); one that returns the document unchanged covers nothing.
+- **Also blocks until a migration names it:** a TTL added, moved to another column or
+  shortened (`covers: { table, ttl: { field, after } }`), because the next sweep deletes
+  every row already past the cutoff; the blocked message counts those rows when
+  `lunora deploy` can read the worker. Refresh the column on the rows that must survive
+  in `up`.
+- **Never blocks, may warn:** a dropped table (its rows stay in place, unreachable),
+  a dropped relation, a TTL lengthened or removed (every row stays), `.commitOrdered()`
+  turned off, and a dropped index. A dropped index is refused only while a function
+  still names it through `.withIndex("name")`; point the query elsewhere first.
+
+A migration covers a change only if it is new since the baseline, its `table`
+is a string literal naming the affected table, and its `up` does more than return
+the document (the `migrate create` placeholder covers nothing). The baseline advances
+only after a successful deploy.
+
+## Online data migrations
+
+A migration transforms one document at a time, runs inside each shard's Durable
+Object in keyset batches, and is resumable: per-shard progress is kept in a
+reserved `__lunora_migrations` table.
+
+Scaffold it rather than writing the file from scratch:
 
-## When to Use
+```bash
+lunora migrate create backfill-display-name --table users
+```
 
-- Adding required fields to existing tables.
-- Changing field types or structure.
-- Splitting/merging tables, renaming/removing fields.
-- Reshaping `.global()` (D1-backed) tables.
+This appends to `lunora/migrations.ts`. Codegen only discovers
+`export const x = defineMigration({...})` with a literal `id`. An
+`export default` is silently skipped, and `lunora migrate up` then can't find it.
 
-## When Not to Use
+```ts
+// lunora/migrations.ts
+import { defineMigration } from "@lunora/server"; // or "lunorash/server"
 
-- Greenfield schema with no data at rest.
-- Adding **optional** fields that need no backfill.
-- Adding new tables or indexes with no correctness concern.
+export const backfillDisplayName = defineMigration({
+    id: "backfill-display-name",
+    table: "users",
+    batchSize: 200, // optional
+    up: (doc) => {
+        if (typeof doc.displayName === "string") {
+            return undefined; // skip: not rewritten, not counted as changed
+        }
 
-## Two Storage Layers — Know Which You Are Migrating
+        return { ...doc, displayName: doc.name ?? "Anonymous" };
+    },
+    // optional; applied by `lunora migrate down`
+    down: (doc) => {
+        const { displayName, ...rest } = doc;
 
-Lunora tables live in one of two backends, and they migrate differently:
+        return rest;
+    },
+});
+```
 
-- **ShardDO SQLite (default `root`, and `.shardBy(key)` tables).** State lives in
-  the per-app / per-shard Durable Object. Data is reshaped with **online data
-  migrations** — `defineMigration` declarations run by `lunora migrate up`,
-  resumable per shard.
-- **`.global()` tables (D1).** Replicated to D1 for cross-region reads. Their
-  structural DDL gets versioned **SQL migrations** via `lunora migrate generate`,
-  applied by `@lunora/d1`'s runner at deploy time.
+The runner keeps each row's original `_id` and `_creationTime`.
 
-A breaking change to a `.global()` table needs a generated SQL migration; a data
-backfill (either layer) is an online `defineMigration`. Both follow the same
-**widen → migrate → narrow** discipline.
+### Reading another table
 
-## Key Principle: Widen, Migrate, Narrow
+The transform's second argument is a shard-scoped, read-only `ctx.db`
+(`get`, `findFirst`, `findMany`, `count`), and the transform may be `async`.
+That covers the common case of copying a field down from a parent:
 
-The schema-drift gate (and D1 itself) will not let a breaking change deploy
-without an accompanying migration. So every breaking change is staged:
+```ts
+up: async (doc, ctx) => {
+    const thread = await ctx.db.get("threads", String(doc.threadId));
 
-1. **Widen** — make the schema accept both old and new shapes (add the new field
-   as `v.optional`, keep the old one). Update reads to handle both; start writing
-   the new shape for new rows. Deploy.
-2. **Migrate** — backfill existing rows to the new shape (an online
-   `defineMigration` run with `lunora migrate up`; plus `lunora migrate generate`
-   for `.global()` structural DDL). Verify completeness with `lunora migrate
-status`.
-3. **Narrow** — make the field required / drop the old field, remove the
-   both-shapes read code. Deploy.
+    return thread ? { ...doc, userId: thread.userId } : undefined;
+},
+```
 
-### Prefer new fields over changing types
+To change rows in a second table, write a second migration over that table.
 
-When changing a field's shape, add a new field rather than mutating the existing
-one — safer transition, easier rollback.
+### A shard key can't be backfilled
 
-### Don't delete data prematurely
+If the field is the table's `.shardBy()` key, a migration can't help: a row with
+no shard key belongs to no shard (so the runner never sees it), and setting the
+key would mean moving the row to another Durable Object. Re-key with
+`lunora export` → rewrite the NDJSON → `lunora import`. A rewrite that adds, drops or splits rows must also update the last line, the end marker `{"end":true,"rows":N}`, to the new row count, or the import refuses the file. Ids are preserved, so
+foreign keys survive. The same route applies to any shard-mode change.
 
-Prefer deprecating: mark the old field `v.optional` with a `// deprecated:` code
-comment explaining why it existed. Delete only once you are sure nothing reads
-it.
+### Run it
 
-## Safe Changes (No Migration Needed)
+```bash
+lunora migrate up backfill-display-name --dry-run   # preview, rewrites nothing
+lunora migrate up backfill-display-name             # against the dev worker (localhost:8787)
+lunora migrate status backfill-display-name         # per-shard progress
+lunora migrate down backfill-display-name           # only if `down` is defined
+```
 
-```ts
-// Adding an optional field — safe.
-users: defineTable({
-    name: v.string(),
-    bio: v.optional(v.string()),
-});
+Options: `--batch-size <n>`, `--steps <n>` (cap batches per run), `--format json`.
 
-// Adding a new table — safe.
-posts: defineTable({ userId: v.id("users"), title: v.string() }).index("by_user", ["userId"]);
+Production runs rewrite live data, so they require an explicit confirmation:
 
-// Adding an index — safe.
-users: defineTable({ name: v.string(), email: v.string() }).index("by_email", ["email"]);
+```bash
+LUNORA_ADMIN_TOKEN=… lunora migrate up backfill-display-name --prod --url https://app.example.com --yes
 ```
 
-## Online Data Migrations (the backfill workhorse)
+`--url` can be omitted when the checkout is linked to production. Alternatively,
+`lunora deploy --migrate --migrate-yes --migrate-url <url>` runs every declared
+migration right after the deploy (`up` is idempotent, so finished ones no-op).
 
-For backfilling/reshaping rows, declare a migration with `defineMigration` from
-`@lunora/server`. It transforms one document at a time, runs **inside each
-shard's** Durable Object in keyset batches, and is **resumable** — per-shard
-progress is tracked in a reserved `__lunora_migrations` table, so an interrupted
-run resumes where it stopped. Codegen discovers declarations and emits them into
-the registry the DO and CLI look up by `id`.
+## `.global()` tables
 
-```ts
-// lunora/migrations/backfill-display-name.ts
-import { defineMigration } from "@lunora/server";
+The runtime provisions `.global()` tables itself on first use, on both D1 and
+Hyperdrive: `CREATE TABLE IF NOT EXISTS`, additive `ADD COLUMN`, and declared
+indexes. Additive changes therefore need no DDL step.
 
-export default defineMigration({
-    id: "backfill-display-name",
-    table: "users",
-    batchSize: 200, // optional; defaults to the runner's batch size
-    up: (doc) => {
-        if (typeof doc.displayName === "string") {
-            return; // already migrated — return undefined to skip (not counted as changed)
-        }
-        return { ...doc, displayName: doc.name ?? "Anonymous" };
-    },
-    // optional reverse transform applied by `lunora migrate down`
-    down: (doc) => {
-        const { displayName, ...rest } = doc as Record<string, unknown>;
-        return rest;
-    },
-});
-```
+For **D1-backed** global tables, `lunora migrate generate` diffs the schema
+against `lunora/migrations/.snapshot.json` and writes a reviewable
+`lunora/migrations/<timestamp>_<name>.sql`:
 
-The transform must preserve row identity — the runner always keeps the original
-`_id` / `_creationTime`, so do not change them.
+```bash
+lunora migrate generate --name add_user_status
+```
 
-### Run it
+No command applies that file, `lunora deploy` included. Its value is the
+statements the runtime never issues (`DROP TABLE`, `DROP INDEX`) plus a comment
+block of changes it couldn't generate (renames, backfilling `UPDATE`s), which you
+write yourself. Apply it deliberately, after review:
 
 ```bash
-lunora migrate create backfill-display-name   # scaffold the migration file
-lunora codegen                                 # discover + register it
-lunora migrate up backfill-display-name --dry-run   # preview, no rows rewritten
-lunora migrate up backfill-display-name        # run across shards (keyset batches)
-lunora migrate status backfill-display-name    # per-shard progress
-lunora migrate down backfill-display-name      # revert (if `down` defined)
+wrangler d1 execute DB --remote --file=lunora/migrations/<file>.sql
 ```
 
-Useful flags: `--batch-size <n>`, `--steps <n>` (cap batches this run), and
-`--prod --url <worker> --yes` to target production (with `LUNORA_ADMIN_TOKEN`).
+The file is multi-statement, so it is not a `@lunora/d1` `Migration`. Commit
+`.snapshot.json` with it, or the next diff won't see a dropped table.
 
-## `.global()` (D1) Structural Migration Flow
+Hyperdrive-backed tables are left out of `migrate generate` (it emits SQLite
+DDL). Write destructive DDL for them by hand against your database. See
+`lunora-setup-hyperdrive-global`.
 
-```bash
-# 1. Edit lunora/schema.ts (widen: add the optional new field to the .global() table).
-lunora codegen
+`defineMigration` does not reach `.global()` rows on either backend: it runs
+inside each shard's Durable Object against that shard's SQLite, never D1 or
+Hyperdrive. Backfill a global table with a SQL `UPDATE` (D1: `wrangler d1
+execute`; Hyperdrive: your Postgres/MySQL client). The drift gate still keys on
+a migration naming the table, so once the SQL backfill is done, deploy the
+narrowing change with `--allow-schema-drift` rather than a no-op migration.
 
-# 2. Generate the SQL migration by diffing schema against the snapshot baseline.
-lunora migrate generate --name=add_user_status
+### Moving a dataset from D1 to Hyperdrive
 
-# Writes lunora/migrations/<timestamp>_add_user_status.sql and updates
-# lunora/migrations/.snapshot.json. Review the SQL before committing.
-
-# 3. Deploy — @lunora/d1's runner applies pending migrations.
-lunora deploy
+```bash
+lunora migrate d1-to-hyperdrive --from-url https://old-worker --to-url https://new-worker
 ```
 
-`lunora migrate generate` only considers `.global()` tables (root/sharded tables
-are not D1-backed). Run it after each schema edit in the widen and narrow steps;
-backfill data with an online migration between them.
-
-## The Schema-Drift Gate
-
-`lunora deploy` (and `verify` / `prepare`) run a **pre-deploy schema-drift gate**:
-it compares the committed structural baseline (`lunora/.lunora-schema.json`)
-against the snapshot codegen produced this run. Breaking drift **without an
-accompanying data migration blocks the deploy**. The baseline is only re-blessed
-_after_ the deploy succeeds, so a failed deploy never advances it past a change
-that never shipped.
-
-If the gate blocks you: that is the signal to stage the change (widen first) or
-add the migration — not to bypass it.
-
-## Common Pitfalls
-
-1. **Making a field required before backfilling.** The drift gate / D1 rejects
-   the deploy because existing rows lack it. Widen first.
-2. **Reshaping rows by hand instead of `defineMigration`.** A hand-rolled
-   `internalMutation` that `.collect()`s a large table hits transaction limits
-   and is not resumable. Use `defineMigration` — it batches and tracks per-shard
-   progress.
-3. **Not writing the new shape during the migration window.** Rows created mid-
-   migration get missed, leaving unmigrated data after it "completes." Start
-   dual-writing in the widen step.
-4. **Skipping the dry run.** `lunora migrate up <id> --dry-run` validates the
-   transform before it touches real rows.
-5. **Deleting a field prematurely.** Deprecate with `v.optional` + a comment;
-   delete only once nothing references it.
-6. **Migrating the wrong layer.** A `.global()` structural change needs `lunora
-migrate generate` (SQL); a data backfill needs a `defineMigration`. Check the
-   table's `.global()` / `.shardBy()` modifier first.
+`--tables a,b` limits it to some tables, and `--out <path>` keeps the
+intermediate NDJSON dump. `lunora migrate hyperdrive-to-d1` takes the same
+flags and moves the data back, for a rollback. Both are insert-only by default:
+a row whose `_id` already exists in the target is reported as a conflict and
+left unchanged, so a rollback restores missing rows, not later edits. Add
+`--replace` (with `--tables` naming the `.global()` tables; it is refused
+without one) to overwrite those rows with the source's values, or keep the dump
+with `--out` and reconcile by hand. This moves the data to another backend; any
+reshaping still follows widen → migrate → narrow.
+
+## Common pitfalls
+
+1. **Making a field required before backfilling.** The gate blocks it unless a
+   new migration on that table is part of the same deploy. Widen first.
+2. **Backfilling with a hand-written mutation.** A mutation that `.collect()`s
+   a large table runs into transaction limits and can't resume.
+   `defineMigration` batches per shard and resumes.
+3. **Not dual-writing during the window.** Rows inserted mid-migration keep the
+   old shape. Write the new shape from the widen step on.
+4. **A non-literal `table`.** `table: TABLE_NAME` covers nothing in the gate.
+5. **Bypassing the gate.** `--allow-schema-drift` ships the shape for one run
+   without checking anything. `lunora deploy` reads the stored rows of the affected tables and
+   lets a change through only when every row already fits.
 
 ## Checklist
 
-- [ ] Identified the change and which layer it touches (ShardDO vs `.global()`).
-- [ ] Widened the schema to accept both shapes; `lunora codegen` clean.
-- [ ] Updated reads to handle both shapes; started writing the new shape.
-- [ ] Deployed the widened schema.
-- [ ] Authored a `defineMigration`; previewed with `lunora migrate up --dry-run`.
-- [ ] Ran `lunora migrate up`; `lunora migrate generate` + deploy for `.global()`
-      structural changes.
-- [ ] Verified completion with `lunora migrate status`.
-- [ ] Narrowed the schema (required / drop old field); removed both-shapes code.
-- [ ] Deployed the final schema; schema-drift gate passed.
+- [ ] Identified the layer (root / `.shardBy()` / `.global()` D1 / Hyperdrive).
+- [ ] Widened the schema; reads handle both shapes; writes use the new shape. Deployed.
+- [ ] `lunora migrate create <id> --table <t>`; transform filled in; `--dry-run` looks right.
+- [ ] `lunora migrate up <id>` (prod: `--prod --yes`); `lunora migrate status <id>` complete on every shard.
+- [ ] D1 destructive DDL: `migrate generate` output reviewed and applied with `wrangler d1 execute`.
+- [ ] Narrowed the schema and removed both-shapes code; drift gate green on deploy.