cfsa-antigravity 2.0.0 → 2.2.0

package/template/.agent/skills/migration-management/SKILL.md

@@ -1,384 +1,200 @@
 ---
 name: database-migration-management
-description: Manage database migrations and schema versioning. Use when planning migrations, version control, rollback strategies, or data transformations in PostgreSQL and MySQL.
+description: Manage database migrations and schema versioning. Use when planning migrations, version control, rollback strategies, or data transformations for any database paradigm.
 ---
 
 # Database Migration Management
 
 ## Overview
 
-Implement robust database migration systems with version control, rollback capabilities, and data transformation strategies. Includes migration frameworks and production deployment patterns.
+Implement robust database migration systems with version control, rollback capabilities, and data transformation strategies. This skill covers universal migration methodology. For paradigm-specific tooling, see references.
 
 ## When to Use
 
 - Schema versioning and evolution
 - Data transformations and cleanup
-- Adding/removing tables and columns
+- Adding/removing tables, columns, collections, or fields
 - Index creation and optimization
 - Migration testing and validation
 - Rollback planning and execution
 - Multi-environment deployments
 
-## Migration Framework Setup
-
-### PostgreSQL - Schema Versioning
-
-```sql
--- Create migrations tracking table
-CREATE TABLE schema_migrations (
-  version BIGINT PRIMARY KEY,
-  name VARCHAR(255) NOT NULL,
-  executed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
-  duration_ms INTEGER,
-  checksum VARCHAR(64)
-);
-
--- Create migration log table
-CREATE TABLE migration_logs (
-  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
-  version BIGINT NOT NULL,
-  status VARCHAR(20) NOT NULL,
-  error_message TEXT,
-  rolled_back_at TIMESTAMP,
-  executed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
-);
-
--- Function to record migration
-CREATE OR REPLACE FUNCTION record_migration(
-  p_version BIGINT,
-  p_name VARCHAR,
-  p_duration_ms INTEGER
-) RETURNS void AS $$
-BEGIN
-  INSERT INTO schema_migrations (version, name, duration_ms)
-  VALUES (p_version, p_name, p_duration_ms)
-  ON CONFLICT (version) DO UPDATE
-  SET executed_at = CURRENT_TIMESTAMP;
-END;
-$$ LANGUAGE plpgsql;
-```
-
-### MySQL - Migration Tracking
-
-```sql
--- Create migrations table for MySQL
-CREATE TABLE schema_migrations (
-  version BIGINT PRIMARY KEY,
-  name VARCHAR(255) NOT NULL,
-  executed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
-  duration_ms INT,
-  checksum VARCHAR(64)
-) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
-
--- Migration status table
-CREATE TABLE migration_status (
-  id INT AUTO_INCREMENT PRIMARY KEY,
-  version BIGINT NOT NULL,
-  status ENUM('pending', 'completed', 'failed', 'rolled_back'),
-  error_message TEXT,
-  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
-) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
-```
+## Paradigm-Specific References
 
-## Common Migration Patterns
-
-### Adding Columns
-
-**PostgreSQL - Safe Column Addition:**
-
-```sql
--- Migration: 20240115_001_add_phone_to_users.sql
-
--- Add column with default (non-blocking)
-ALTER TABLE users
-ADD COLUMN phone VARCHAR(20) DEFAULT '';
-
--- Add constraint after population
-ALTER TABLE users
-ADD CONSTRAINT phone_format
-CHECK (phone = '' OR phone ~ '^\+?[0-9\-\(\)]{10,}$');
-
--- Create index
-CREATE INDEX CONCURRENTLY idx_users_phone ON users(phone);
-
--- Rollback:
--- DROP INDEX CONCURRENTLY idx_users_phone;
--- ALTER TABLE users DROP COLUMN phone;
-```
-
-**MySQL - Column Addition:**
+After reading the methodology below, read the reference matching your surface's Databases column in the surface stack map (`.agent/instructions/tech-stack.md`):
 
-```sql
--- Migration: 20240115_001_add_phone_to_users.sql
+| Paradigm | Reference | Example Stores |
+|----------|-----------|----------------|
+| Relational | `references/relational.md` | PostgreSQL, MySQL, SQLite |
+| Document | `references/document.md` | MongoDB, Firestore, CouchDB |
+| Graph | `references/graph.md` | SurrealDB, Neo4j, ArangoDB |
 
--- Add column with ALTER
-ALTER TABLE users
-ADD COLUMN phone VARCHAR(20) DEFAULT '',
-ADD INDEX idx_phone (phone);
-
--- Rollback:
--- ALTER TABLE users DROP COLUMN phone;
-```
-
-### Renaming Columns
-
-**PostgreSQL - Column Rename:**
-
-```sql
--- Migration: 20240115_002_rename_user_name_columns.sql
-
--- Rename columns
-ALTER TABLE users RENAME COLUMN user_name TO full_name;
-ALTER TABLE users RENAME COLUMN user_email TO email_address;
-
--- Update indexes
-REINDEX TABLE users;
+---
 
--- Rollback:
--- ALTER TABLE users RENAME COLUMN email_address TO user_email;
--- ALTER TABLE users RENAME COLUMN full_name TO user_name;
-```
+## Universal Migration Principles
 
-### Creating Indexes Non-blocking
+### 1. Bidirectional Migrations
 
-**PostgreSQL - Concurrent Index Creation:**
+Every migration must have both directions:
+- **UP** — applies the change
+- **DOWN** — reverses the change
 
-```sql
--- Migration: 20240115_003_add_performance_indexes.sql
+If a migration cannot be reversed (e.g., data deletion), document this explicitly and require a backup before execution.
 
--- Create indexes without blocking writes
-CREATE INDEX CONCURRENTLY idx_orders_user_created
-ON orders(user_id, created_at DESC);
+### 2. Migration Ordering
 
-CREATE INDEX CONCURRENTLY idx_products_category_active
-ON products(category_id)
-WHERE active = true;
+Migrations are **sequentially ordered** by timestamp or version number:
+- Timestamp format: `YYYYMMDD_NNN_description` (e.g., `20240115_001_add_phone_to_users`)
+- Never reorder or renumber existing migrations
+- Never modify an already-executed migration — create a new one
 
--- Verify index creation
-SELECT schemaname, tablename, indexname, idx_scan
-FROM pg_stat_user_indexes
-WHERE indexname LIKE 'idx_%';
+### 3. Idempotency
 
--- Rollback:
--- DROP INDEX CONCURRENTLY idx_orders_user_created;
--- DROP INDEX CONCURRENTLY idx_products_category_active;
-```
+Migrations should be idempotent where possible:
+- Use "IF NOT EXISTS" / "IF EXISTS" guards
+- Check for the presence of the change before applying it
+- Run pre-flight checks before destructive operations
 
-**MySQL - Online Index Creation:**
+### 4. Atomic Migrations
 
-```sql
--- Migration: 20240115_003_add_performance_indexes.sql
+Each migration should be a single logical change:
+- One migration = one purpose (add column, create table, transform data)
+- Don't combine unrelated changes
+- Use transactions for multi-statement migrations where supported
 
--- Create indexes with ALGORITHM=INPLACE and LOCK=NONE
-ALTER TABLE orders
-ADD INDEX idx_user_created (user_id, created_at),
-ALGORITHM=INPLACE, LOCK=NONE;
+---
 
--- Monitor progress
-SELECT * FROM INFORMATION_SCHEMA.PROCESSLIST
-WHERE INFO LIKE 'ALTER TABLE%';
-```
+## Common Migration Patterns
 
-### Data Transformations
+### Safe Schema Changes
 
-**PostgreSQL - Data Cleanup Migration:**
+| Operation | Safe? | Strategy |
+|-----------|-------|----------|
+| Add nullable column | ✅ Always safe | Add with default or NULL |
+| Add NOT NULL column | ⚠️ Requires backfill | Add nullable → backfill → add constraint |
+| Rename column | ⚠️ Requires coordination | Add new → copy data → update app → drop old |
+| Drop column | ⚠️ Requires coordination | Stop reading → deploy → drop column |
+| Add index | ⚠️ Can lock table | Use concurrent/online index creation |
+| Drop table | 🔴 Destructive | Backup first, confirm no references |
 
-```sql
--- Migration: 20240115_004_normalize_email_addresses.sql
+### Multi-Phase Migrations
 
--- Normalize existing email addresses
-UPDATE users
-SET email = LOWER(TRIM(email))
-WHERE email != LOWER(TRIM(email));
+For non-trivial schema changes (rename, type change, restructure):
 
--- Remove duplicates by keeping latest
-DELETE FROM users
-WHERE id NOT IN (
-  SELECT DISTINCT ON (LOWER(email)) id
-  FROM users
-  ORDER BY LOWER(email), created_at DESC
-);
+1. **Phase 1: Expand** — Add new structure alongside old
+2. **Phase 2: Migrate** — Copy/transform data to new structure
+3. **Phase 3: Contract** — Remove old structure after verification
 
--- Rollback: Restore from backup (no safe rollback for data changes)
-```
+This pattern prevents downtime and allows rollback at any phase.
 
-**MySQL - Bulk Data Update:**
+### Data Transformation Migrations
 
-```sql
--- Migration: 20240115_004_update_product_categories.sql
+When transforming data (normalization, cleanup, format changes):
 
--- Update multiple rows with JOIN
-UPDATE products p
-JOIN category_mapping cm ON p.old_category = cm.old_name
-SET p.category_id = cm.new_category_id
-WHERE p.old_category IS NOT NULL;
+1. **Verify source data** — Count records, check constraints
+2. **Apply transformation** — Batch for large datasets
+3. **Validate results** — Compare counts, spot-check values
+4. **Document irreversibility** — If rollback requires backup restore, say so
 
--- Verify update
-SELECT COUNT(*) as updated_count
-FROM products
-WHERE category_id IS NOT NULL;
-```
+---
 
-### Table Structure Changes
+## Rollback Strategies
 
-**PostgreSQL - Alter Table Migration:**
+### Per-Migration Rollback
 
-```sql
--- Migration: 20240115_005_modify_order_columns.sql
+Every migration has a DOWN direction. Execute in reverse order:
+- Migration 5 DOWN
+- Migration 4 DOWN
+- Migration 3 DOWN
+- ... until target version reached
 
--- Add new column
-ALTER TABLE orders
-ADD COLUMN status_updated_at TIMESTAMP;
+### Point-in-Time Recovery
 
--- Add constraint
-ALTER TABLE orders
-ADD CONSTRAINT valid_status
-CHECK (status IN ('pending', 'processing', 'completed', 'cancelled'));
+For data-destructive migrations:
+- Require backup before execution
+- Document the backup location
+- Test restore procedure before migration
 
--- Set default for existing records
-UPDATE orders
-SET status_updated_at = updated_at
-WHERE status_updated_at IS NULL;
+### Blue-Green Schema Changes
 
--- Make column NOT NULL
-ALTER TABLE orders
-ALTER COLUMN status_updated_at SET NOT NULL;
+For zero-downtime migrations:
+- Run new schema alongside old
+- Application supports both schemas during transition
+- Cut over when new schema is verified
+- Drop old schema after observation period
 
--- Rollback:
--- ALTER TABLE orders DROP COLUMN status_updated_at;
--- ALTER TABLE orders DROP CONSTRAINT valid_status;
-```
+---
 
 ## Testing Migrations
 
-**PostgreSQL - Test in Transaction:**
-
-```sql
--- Test migration in transaction (will be rolled back)
-BEGIN;
-
--- Run migration statements
-ALTER TABLE users ADD COLUMN test_column VARCHAR(255);
+### Local Testing
 
--- Validate data
-SELECT COUNT(*) FROM users;
-SELECT COUNT(DISTINCT email) FROM users;
+1. **Fresh database** — Run all migrations from scratch (empty → current)
+2. **Incremental** — Run only new migrations on a copy of production
+3. **Rollback** — Test DOWN migration after UP
+4. **Data integrity** — Verify constraints hold after migration
 
--- Rollback if issues found
-ROLLBACK;
+### Staging Validation
 
--- Or commit if all good
-COMMIT;
-```
+1. Restore production backup to staging
+2. Run migrations on staging
+3. Verify application works against migrated schema
+4. Run automated tests against staging
+5. Record migration timing for capacity planning
 
-**Validate Migration:**
+### Pre-Production Checklist
 
-```sql
--- Check migration was applied
-SELECT version, name, executed_at FROM schema_migrations
-WHERE version = 20240115005;
+- [ ] Tested on local database
+- [ ] Tested on staging with production data copy
+- [ ] Rollback procedure tested
+- [ ] Backup verified before migration
+- [ ] Application code compatible with both old and new schema
+- [ ] Migration timing measured (acceptable for maintenance window?)
+- [ ] Long-running query impact assessed
 
--- Verify table structure
-SELECT column_name, data_type, is_nullable
-FROM information_schema.columns
-WHERE table_name = 'users'
-ORDER BY ordinal_position;
-```
-
-## Rollback Strategies
-
-**PostgreSQL - Bidirectional Migrations:**
-
-```sql
--- Migration file: 20240115_006_add_user_status.sql
-
--- ===== UP =====
-CREATE TYPE user_status AS ENUM ('active', 'suspended', 'deleted');
-ALTER TABLE users ADD COLUMN status user_status DEFAULT 'active';
-
--- ===== DOWN =====
--- ALTER TABLE users DROP COLUMN status;
--- DROP TYPE user_status;
-```
-
-**Rollback Execution:**
-
-```sql
--- Function to rollback to specific version
-CREATE OR REPLACE FUNCTION rollback_to_version(p_target_version BIGINT)
-RETURNS TABLE (version BIGINT, name VARCHAR, status VARCHAR) AS $$
-BEGIN
-  -- Execute down migrations in reverse order
-  RETURN QUERY
-  SELECT m.version, m.name, 'rolled_back'::VARCHAR
-  FROM schema_migrations m
-  WHERE m.version > p_target_version
-  ORDER BY m.version DESC;
-END;
-$$ LANGUAGE plpgsql;
-```
+---
 
 ## Production Deployment
 
-**Safe Migration Checklist:**
+### Safe Migration Checklist
 
-- Test migration on production-like database
-- Verify backup exists before migration
-- Schedule during low-traffic window
-- Monitor table locks and long-running queries
-- Have rollback plan ready
-- Test rollback procedure
-- Document all changes
-- Run in transaction when possible
-- Verify data integrity after migration
-- Update application code coordinated with migration
+- [ ] Test migration on production-like database
+- [ ] Verify backup exists before migration
+- [ ] Schedule during low-traffic window
+- [ ] Monitor locks and long-running queries
+- [ ] Have rollback plan ready and tested
+- [ ] Document all changes
+- [ ] Run in transaction when possible
+- [ ] Verify data integrity after migration
+- [ ] Coordinate application code deployment with migration
+- [ ] Notify team of migration window
 
-**PostgreSQL - Long Transaction Safety:**
+### Post-Migration Verification
 
-```sql
--- Use statement timeout to prevent hanging migrations
-SET statement_timeout = '30min';
+1. **Schema check** — Verify new structures exist
+2. **Data check** — Verify data integrity constraints
+3. **Application check** — Verify application reads/writes correctly
+4. **Performance check** — Verify query performance hasn't degraded
+5. **Monitoring** — Watch error rates for 24 hours post-migration
 
--- Use lock timeout to prevent deadlocks
-SET lock_timeout = '5min';
-
--- Run migration with timeouts
-ALTER TABLE large_table
-ADD COLUMN new_column VARCHAR(255),
-ALGORITHM='INPLACE';
-```
-
-## Migration Examples
-
-**Combined Migration - Multiple Changes:**
-
-```sql
--- Migration: 20240115_007_refactor_user_tables.sql
-
-BEGIN;
-
--- 1. Create new column with data from old column
-ALTER TABLE users ADD COLUMN full_name VARCHAR(255);
-UPDATE users SET full_name = first_name || ' ' || last_name;
-
--- 2. Add indexes
-CREATE INDEX idx_users_full_name ON users(full_name);
+---
 
--- 3. Add new constraint
-ALTER TABLE users
-ADD CONSTRAINT email_unique UNIQUE(email);
+## Migration Tracking
 
--- 4. Drop old columns (after verification)
--- ALTER TABLE users DROP COLUMN first_name;
--- ALTER TABLE users DROP COLUMN last_name;
+Maintain a migration history table (or equivalent) that records:
+- Migration version/ID
+- Migration name/description
+- Execution timestamp
+- Duration
+- Checksum (to detect tampering)
+- Status (applied, rolled back, failed)
 
-COMMIT;
-```
+---
 
 ## Resources
 
-- [Flyway - Java Migration Tool](https://flywaydb.org/)
-- [Liquibase - Database Changelog](https://www.liquibase.org/)
-- [Alembic - Python Migration](https://alembic.sqlalchemy.org/)
-- [PostgreSQL ALTER TABLE](https://www.postgresql.org/docs/current/sql-altertable.html)
-- [MySQL ALTER TABLE](https://dev.mysql.com/doc/refman/8.0/en/alter-table.html)
+- [Flyway](https://flywaydb.org/) — Java/JVM migration tool
+- [Liquibase](https://www.liquibase.org/) — Database changelog management
+- [Alembic](https://alembic.sqlalchemy.org/) — Python/SQLAlchemy migrations
+- [Knex.js](https://knexjs.org/) — Node.js query builder with migrations
+- [golang-migrate](https://github.com/golang-migrate/migrate) — Go migration tool