@jamie-tam/forge 6.0.0

package/skills/deliver-db-migration/SKILL.md

@@ -0,0 +1,466 @@
+---
+name: deliver-db-migration
+description: "Use when a change adds, alters, or drops database schema elements or edits ORM schema files (Prisma, Drizzle, Alembic, etc.)."
+---
+
+# Deliver: Database Migration
+
+## Overview
+
+Migrations can corrupt data or cause downtime. This skill enforces generate → test forward → test rollback → test forward again → verify data integrity, before deploy.
+
+**Core Principle:** Every migration is reversible until proven otherwise. If you cannot write a rollback, you cannot deploy the migration.
+
+## The Iron Laws of Database Migration
+
+```
+Every migration MUST have a rollback script.
+NEVER modify a migration that has been deployed to any environment.
+ALWAYS test rollback before deploying the forward migration.
+NEVER run destructive operations (DROP, TRUNCATE) without explicit user confirmation.
+NEVER assume seed data is idempotent -- verify it.
+```
+
+## When to Use
+
+- Architecture artifacts include database schema changes
+- A feature requires new tables, columns, indexes, or constraints
+- An existing schema needs modification (rename, type change, removal)
+- Data needs to be transformed as part of a schema change
+- Database needs to be seeded with initial data
+
+**Do NOT use when:**
+- Application code changes only (no schema changes)
+- Query optimization only (no DDL changes)
+- Read-only reporting changes
+
+## I/O Contract
+
+| Field | Value |
+|---|---|
+| **Requires** | DB schema from `aiwiki/architecture/db-schema.md` (prototype-driven flow, written by harden) OR `.forge/work/{type}/{name}/architecture/db-schema.md` (non-prototype fallback, written by plan-architecture) |
+| **Produces** | Migration files (forward + rollback) in project's migration directory |
+| **Feeds into** | `deliver-deploy` (migrations run during deployment) |
+| **Updates** | `.forge/work/{type}/{name}/manifest.yaml`: migration status, file paths |
+
+### Input Verification
+
+Before starting, read and verify:
+
+```
+.forge/work/{type}/{name}/
+  architecture/
+    db-schema.md         -> Must contain the target schema state
+  manifest.yaml          -> Must exist, feature in progress
+```
+
+Also verify:
+- Project's migration tool is identified (Prisma, Knex, Alembic, Flyway, Django, TypeORM, etc.)
+- Database connection is available for testing
+- Current schema state is known (check existing migrations)
+
+### Output Artifacts
+
+```
+{project-migration-directory}/
+  {timestamp}_{description}.{ext}          -> Forward migration
+  {timestamp}_{description}_rollback.{ext} -> Rollback migration (or inline down() method)
+
+.forge/work/{type}/{name}/
+  manifest.yaml                            -> Updated with migration status
+```
+
+## Migration Tool Detection
+
+Detect the project's migration framework from project structure:
+
+| Indicator | Framework | Migration Directory |
+|---|---|---|
+| `prisma/schema.prisma` | Prisma | `prisma/migrations/` |
+| `knexfile.js` / `knexfile.ts` | Knex.js | `migrations/` |
+| `alembic.ini` / `alembic/` | Alembic (SQLAlchemy) | `alembic/versions/` |
+| `flyway.conf` / `flyway/` | Flyway | `sql/` or `flyway/sql/` |
+| `manage.py` + Django | Django | `{app}/migrations/` |
+| `ormconfig.json` / TypeORM config | TypeORM | `src/migrations/` |
+| `sequelize` in package.json | Sequelize | `migrations/` |
+| `drizzle.config.ts` | Drizzle | `drizzle/` |
+| `src/db/migrations/` exists | Custom/raw SQL | `src/db/migrations/` |
+| `supabase/` directory | Supabase | `supabase/migrations/` |
+
+If no migration tool is detected, ASK the user. Do not guess.
+
+## The Migration Pipeline
+
+### Phase 1: Schema Diff Analysis
+
+**Purpose:** Understand exactly what needs to change, and classify the risk level.
+
+1. **Read Current Schema State**
+   - Examine existing migrations to understand current state
+   - If ORM has introspection (e.g., `prisma db pull`), use it
+   - Document the "before" state clearly
+
+2. **Read Target Schema State**
+   - From `aiwiki/architecture/db-schema.md` (prototype-driven flow, written by harden) OR `.forge/work/{type}/{name}/architecture/db-schema.md` (non-prototype fallback, written by plan-architecture)
+   - This is the "after" state
+
+3. **Compute the Diff**
+   - List every change: new tables, new columns, modified columns, removed columns, new indexes, new constraints, removed elements
+   - For each change, classify:
+
+   | Risk Level | Operations | Handling |
+   |---|---|---|
+   | **Safe** | ADD column (nullable), ADD table, ADD index | Proceed normally |
+   | **Moderate** | ADD column (NOT NULL with default), RENAME column, ADD constraint | Test with existing data |
+   | **Dangerous** | DROP column, DROP table, CHANGE column type, TRUNCATE | Require explicit user confirmation |
+   | **Critical** | DROP database, DROP multiple tables, data type narrowing | Double confirmation, mandatory backup |
+
+4. **Flag Dangerous Operations**
+   ```
+   MIGRATION RISK ASSESSMENT
+   -------------------------
+   [SAFE]      ADD TABLE: payments
+   [SAFE]      ADD COLUMN: users.stripe_customer_id (VARCHAR, nullable)
+   [MODERATE]  ADD COLUMN: orders.total_amount (DECIMAL, NOT NULL, default 0.00)
+   [DANGEROUS] DROP COLUMN: orders.legacy_price
+               -> Contains data in 4,231 rows. Requires confirmation.
+   [DANGEROUS] CHANGE TYPE: products.price FROM INTEGER TO DECIMAL
+               -> Potential precision changes. Verify data conversion.
+
+   ACTION REQUIRED: 2 dangerous operations need user confirmation before proceeding.
+   ```
+
+### Phase 1.5: Codex Mode Check
+
+Now that the schema diff and risk classification are known, run the Codex consent flow from `protocols/codex.md`. The selected mode applies for the rest of this skill's invocation.
+
+- **Takeover:** Dispatch Codex with the schema diff, target schema, and existing migration conventions to generate forward migration, rollback, and migration README. Claude reviews, then runs the 5-step test sequence (Phase 4).
+- **Verify** or **Skip / Codex unavailable:** Proceed with Phases 2-3 below to generate migrations manually. Codex will review Claude's generated migrations before Phase 4 (Verify only).
+
+### Phase 2: Generate Forward Migration
+
+**Purpose:** Create the migration that moves the schema from current state to target state.
+
+1. **Use the Project's Migration Tool**
+   - If tool supports auto-generation (Prisma, Alembic, Django), use it
+   - Review the auto-generated migration for correctness
+   - If tool does not auto-generate, write migration manually
+
+2. **Migration File Standards**
+   - Clear, descriptive name: `20260325_add_payments_table.sql`
+   - Comments explaining WHY each change exists (not just WHAT)
+   - Group related changes logically
+   - Handle NULL/default values explicitly for existing data
+
+3. **Data Migration (when needed)**
+   - If schema change requires data transformation, include it
+   - Example: splitting a `full_name` column into `first_name` + `last_name`
+   - Data migration runs AFTER schema change, BEFORE dropping old column
+   - Data migration MUST be idempotent (safe to run multiple times)
+
+4. **Ordering Matters**
+   ```
+   Correct order for destructive changes:
+   1. Add new columns/tables
+   2. Migrate data from old to new
+   3. Update application code to use new columns
+   4. (After deployment verified) Drop old columns in SEPARATE migration
+   ```
+
+   Prefer multi-step migrations over single destructive ones.
+
+### Phase 3: Generate Rollback Migration
+
+**Purpose:** Create the migration that reverses the forward migration exactly.
+
+1. **Every Forward Operation Has a Reverse**
+
+   | Forward Operation | Rollback Operation |
+   |---|---|
+   | CREATE TABLE | DROP TABLE |
+   | ADD COLUMN | DROP COLUMN |
+   | DROP COLUMN | ADD COLUMN (with data restoration if possible) |
+   | RENAME COLUMN | RENAME COLUMN (reverse) |
+   | ADD INDEX | DROP INDEX |
+   | ADD CONSTRAINT | DROP CONSTRAINT |
+   | ALTER COLUMN TYPE | ALTER COLUMN TYPE (original) |
+   | INSERT seed data | DELETE seed data (by known identifiers) |
+
+2. **Data Preservation in Rollbacks**
+   - If forward migration drops a column, rollback CANNOT restore the data
+   - This must be documented: "ROLLBACK WARNING: Data in column X will not be restored"
+   - Suggest: before dropping columns, create a data backup step
+   - If framework supports it, store dropped column data in a temporary table
+
+3. **Rollback File Standards**
+   - Named to clearly associate with forward migration
+   - Comments explaining what this rolls back and any data loss implications
+   - If rollback is impossible (e.g., lossy type conversion), document explicitly
+
+4. **When Rollback Is Not Possible**
+   - Some operations are truly irreversible (data deletion, lossy type conversion)
+   - Document this clearly in the migration file AND in the deploy plan
+   - Require user acknowledgment: "This migration cannot be rolled back. Data backup is required before proceeding."
+   - Ensure a database backup step is added to the deploy plan
+
+### Phase 3.5: Codex Migration Verify
+
+After both migrations are generated, check the mode recorded at Phase 1.5. If **Verify** was selected, dispatch Codex to review the forward migration, rollback migration, and README for correctness, reversibility, and convention alignment. Address any CRITICAL findings before proceeding to Phase 4. If **Takeover** was selected, skip this step (Codex already generated the migrations). If **Skip**, do nothing. Do NOT re-run the consent flow. See **Codex Integration** section below for full details.
+
+### Phase 4: Test the Migration
+
+**Purpose:** Verify the migration works correctly in all scenarios before deployment.
+
+This is the critical phase. Do NOT skip any step.
+
+#### Test 1: Forward Migration on Clean Database
+
+```
+1. Start with empty database (or known baseline schema)
+2. Run all existing migrations to reach current state
+3. Run the new forward migration
+4. Verify: schema matches target state
+5. Verify: no errors in migration output
+```
+
+#### Test 2: Rollback Migration
+
+```
+1. Starting from state after Test 1 (migration applied)
+2. Run the rollback migration
+3. Verify: schema matches pre-migration state
+4. Verify: no errors in rollback output
+5. Verify: application still works with old schema
+```
+
+#### Test 3: Forward Again (Idempotency Check)
+
+```
+1. Starting from state after Test 2 (rolled back)
+2. Run the forward migration again
+3. Verify: schema matches target state
+4. Verify: migration is idempotent (no "already exists" errors)
+```
+
+#### Test 4: Forward Migration with Seed Data
+
+```
+1. Start with database containing seed/test data
+2. Run the forward migration
+3. Verify: existing data is preserved (count rows before and after)
+4. Verify: data transformations applied correctly
+5. Verify: constraints are satisfied (no orphaned foreign keys)
+6. Verify: no duplicate data created
+```
+
+#### Test 5: Seed Data Idempotency (Docker Environments)
+
+**This is a commonly missed issue in Docker environments:**
+
+```
+1. Run seed script once -> verify data
+2. Run seed script again -> verify NO duplicate data
+3. Run seed script a third time -> verify data is still correct
+
+Common failure: seed script uses INSERT without checking existence,
+causing duplicate rows on container restart.
+
+Fix: Use INSERT ... ON CONFLICT DO NOTHING (Postgres)
+     or INSERT IGNORE (MySQL)
+     or merge/upsert pattern
+```
+
+#### Test Results
+
+```
+MIGRATION TEST RESULTS
+----------------------
+[PASS] Forward migration on clean DB
+[PASS] Rollback migration
+[PASS] Forward migration after rollback (idempotency)
+[PASS] Forward migration with seed data (4,231 rows preserved)
+[PASS] Seed data idempotency (no duplicates after 3 runs)
+[WARN] Rollback will lose data in: orders.legacy_price (acknowledged by user)
+
+MIGRATION READY FOR DEPLOYMENT
+```
+
+### Phase 5: Documentation and Handoff
+
+1. **Update Feature Manifest**
+   ```yaml
+   migration:
+     status: tested
+     forward: "migrations/20260325_add_payments_table.sql"
+     rollback: "migrations/20260325_add_payments_table_rollback.sql"
+     risk_level: moderate
+     data_loss_on_rollback: ["orders.legacy_price"]
+     tested:
+       clean_db: pass
+       rollback: pass
+       idempotency: pass
+       with_data: pass
+       seed_idempotency: pass
+   ```
+
+2. **Migration README (in migration directory)**
+   ```markdown
+   ## Migration: Add Payments Table
+   **Date**: 2026-03-25
+   **Feature**: add-payment-processing
+   **Risk Level**: Moderate
+
+   ### Changes
+   - Creates `payments` table with columns: id, user_id, amount, status, created_at
+   - Adds `stripe_customer_id` column to `users` table (nullable)
+   - Drops `legacy_price` column from `orders` table
+
+   ### Rollback Notes
+   - `orders.legacy_price` data cannot be restored after rollback
+   - Ensure data backup before deploying if this column has needed data
+
+   ### Dependencies
+   - Requires Stripe integration to be configured (but migration is independent)
+   ```
+
+3. **Hand Off to deliver-deploy**
+   - Migration files are ready in the project's migration directory
+   - Deploy skill will execute them as part of the deployment pipeline
+   - Rollback scripts are available if deployment needs to be reversed
+
+## Common Migration Patterns
+
+### Adding a Required Column to Existing Table
+
+**Wrong:**
+```sql
+ALTER TABLE users ADD COLUMN email VARCHAR(255) NOT NULL;
+-- Fails if table has existing rows without email
+```
+
+**Right (two-step):**
+```sql
+-- Migration 1: Add nullable column
+ALTER TABLE users ADD COLUMN email VARCHAR(255);
+
+-- Migration 2 (after backfilling data): Add NOT NULL constraint
+UPDATE users SET email = 'unknown@placeholder.com' WHERE email IS NULL;
+ALTER TABLE users ALTER COLUMN email SET NOT NULL;
+```
+
+### Renaming a Column Safely
+
+**Wrong:**
+```sql
+ALTER TABLE users RENAME COLUMN name TO full_name;
+-- Breaks all application code referencing 'name' immediately
+```
+
+**Right (three-step, zero-downtime):**
+```sql
+-- Migration 1: Add new column, copy data
+ALTER TABLE users ADD COLUMN full_name VARCHAR(255);
+UPDATE users SET full_name = name;
+
+-- Deploy: Update app code to write to both, read from full_name
+
+-- Migration 2 (after app deployed): Drop old column
+ALTER TABLE users DROP COLUMN name;
+```
+
+### Changing Column Type
+
+```sql
+-- Safe approach: add new column, migrate data, swap
+ALTER TABLE products ADD COLUMN price_decimal DECIMAL(10,2);
+UPDATE products SET price_decimal = price::DECIMAL / 100.0;
+
+-- After verification:
+ALTER TABLE products DROP COLUMN price;
+ALTER TABLE products RENAME COLUMN price_decimal TO price;
+```
+
+## Red Flags -- STOP and Reconsider
+
+| Thought | Reality |
+|---|---|
+| "It's just adding a column" | Adding a NOT NULL column to a million-row table can lock it for minutes. |
+| "We don't need rollback for this" | You ALWAYS need rollback. This is not negotiable. |
+| "I'll fix the migration file" | If it has been deployed anywhere, create a new migration instead. |
+| "Seed data will be fine" | Docker restarts will re-run seeds. Test for duplicates. |
+| "The ORM handles rollback" | Verify it. Auto-generated rollbacks are often incomplete. |
+| "DROP is fine, nobody uses that column" | Verify with data queries. "Nobody uses it" is an assumption. |
+| "We can restore from backup" | Restoring a full backup for one column is overkill. Plan properly. |
+| "Testing on my local DB is enough" | Test with production-like data volumes and constraints. |
+
+## Framework-Specific Notes
+
+### Prisma
+- Use `prisma migrate dev` for development, `prisma migrate deploy` for production
+- Prisma auto-generates rollback in the migration SQL
+- Check `_prisma_migrations` table for applied migrations
+- Watch for: Prisma shadow database requirements
+
+### Alembic (SQLAlchemy)
+- Always implement both `upgrade()` and `downgrade()`
+- Use `alembic downgrade -1` to test rollback
+- Use `--autogenerate` but ALWAYS review the output
+- Watch for: Alembic cannot detect table renames (sees drop + create)
+
+### Django
+- Use `makemigrations` then review before `migrate`
+- `migrate {app} {previous_number}` for rollback
+- `RunPython` operations need `reverse_code` parameter
+- Watch for: Django circular migration dependencies
+
+### Knex.js
+- Implement both `up()` and `down()` in migration file
+- Use `knex migrate:rollback` to test
+- Watch for: Knex does not auto-generate migrations
+
+### TypeORM
+- Implement both `up()` and `down()` in migration class
+- Use `typeorm migration:revert` to test rollback
+- Watch for: TypeORM sync mode can bypass migrations (disable in production)
+
+## Codex Integration
+**Modes:** Verify or Takeover | **Protocol:** `protocols/codex.md`
+
+- **Verify:** Claude generates migrations, Codex reviews for correctness.
+- **Takeover:** Codex generates forward + rollback migrations, Claude reviews before testing.
+
+**When:** After Claude has analyzed the schema diff and classified risk (Phase 1), but before generating migration files (Phase 2-3).
+
+**Context to pass:**
+- Path to `.forge/work/{type}/{name}/architecture/db-schema.md` (target schema)
+- Path to current migration files (for convention matching)
+- Schema diff summary from Phase 1
+
+**What Codex generates:**
+- Forward migration file (Phase 2)
+- Rollback migration file (Phase 3)
+- Migration README
+
+**Prompt focus:** "Generate forward and rollback migration files matching the project's migration tool conventions. The forward migration must implement the schema diff. The rollback must reverse every forward operation. Follow the existing migration naming and style conventions."
+
+**Presentation:** Claude reviews generated migrations for correctness, then runs the 5-step test sequence (Phase 4) regardless of who generated the migration.
+
+---
+
+## Integration with Other Skills
+
+- **plan-architecture** produces the `db-schema.md` that this skill consumes
+- **deliver-deploy** executes the migrations this skill produces
+- **support-debug** is invoked if migration testing reveals issues
+- **support-gotcha** records migration lessons (especially the Docker seed issue)
+- **quality-test-execution** verifies application behavior with the new schema
+
+## Quick Reference
+
+| Phase | Key Actions | Gate Criteria |
+|---|---|---|
+| **1. Schema Diff** | Compare current vs target, classify risk | Risk assessment complete, dangerous ops confirmed |
+| **2. Forward Migration** | Generate migration using project's tool | Migration file created, follows standards |
+| **3. Rollback Migration** | Generate reverse migration | Rollback exists for every forward operation |
+| **4. Testing** | 5-step test sequence | All 5 tests pass |
+| **5. Documentation** | Update manifest, create migration README | Manifest updated, handoff to deploy ready |