@weirdfingers/baseboards 0.9.6 → 0.9.7

package/templates/api/docs/MIGRATIONS.md

@@ -1,472 +0,0 @@
-# Database Migrations Guide
-
-This guide covers the complete workflow for making database schema changes in the Boards backend using our SQL DDL-first migration system.
-
-## Overview
-
-The Boards backend uses a **schema-first approach** where:
-
-1. **SQL DDL files** are the single source of truth for database schema
-2. **SQLAlchemy models** are auto-generated from the schema
-3. **Migration scripts** are auto-generated by comparing schemas
-4. **Database changes** are applied via generated migrations
-
-## Quick Reference
-
-```bash
-# Complete workflow
-vim migrations/schemas/002_new_feature.sql  # 1. Edit schema
-python scripts/generate_migration.py --name new_feature  # 2. Generate migration
-psql boards_dev < migrations/generated/*_new_feature_up.sql  # 3. Apply migration
-python scripts/generate_models.py  # 4. Regenerate models
-```
-
-## Detailed Workflow
-
-### Step 1: Modify the SQL DDL Schema
-
-Schema files are stored in `migrations/schemas/` and processed in alphabetical order.
-
-#### Creating a New Schema File
-
-```bash
-cd packages/backend
-
-# Create new schema file with incremental numbering
-vim migrations/schemas/002_add_user_preferences.sql
-```
-
-#### Schema File Example
-
-```sql
--- 002_add_user_preferences.sql
--- Add user preferences and notification settings
-
--- New table for user preferences
-CREATE TABLE user_preferences (
-    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
-    user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
-    theme VARCHAR(20) DEFAULT 'light' CHECK (theme IN ('light', 'dark')),
-    timezone VARCHAR(50) DEFAULT 'UTC',
-    email_notifications BOOLEAN DEFAULT TRUE,
-    push_notifications BOOLEAN DEFAULT FALSE,
-    metadata JSONB DEFAULT '{}',
-    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-    UNIQUE(user_id)
-);
-
--- Indexes for performance
-CREATE INDEX idx_user_preferences_user ON user_preferences(user_id);
-CREATE INDEX idx_user_preferences_theme ON user_preferences(theme);
-
--- Trigger for updated_at
-CREATE TRIGGER update_user_preferences_updated_at
-    BEFORE UPDATE ON user_preferences
-    FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
-
--- Add new column to existing table
-ALTER TABLE boards ADD COLUMN view_count INTEGER DEFAULT 0;
-CREATE INDEX idx_boards_view_count ON boards(view_count);
-```
-
-#### Schema File Best Practices
-
-- **Incremental naming**: `001_`, `002_`, `003_` etc.
-- **Descriptive names**: `add_user_preferences`, `create_analytics_tables`
-- **Comments**: Document the purpose of changes
-- **Idempotent**: Use `IF NOT EXISTS` where appropriate
-- **Constraints**: Add proper constraints and validations
-- **Indexes**: Include necessary indexes for performance
-- **Triggers**: Add triggers for `updated_at` columns
-
-### Step 2: Generate Migration Scripts
-
-Use the migration generator to create UP and DOWN scripts:
-
-```bash
-# Generate migration comparing current DB to target schema
-python scripts/generate_migration.py --name add_user_preferences
-
-# Output files created:
-# migrations/generated/20240830_143022_add_user_preferences_up.sql
-# migrations/generated/20240830_143022_add_user_preferences_down.sql
-```
-
-#### Generated Migration Example
-
-**UP Migration** (`*_up.sql`):
-
-```sql
--- Migration: 20240830_143022_add_user_preferences UP
--- Generated: 2024-08-30T14:30:22.123456
-
-CREATE TABLE "public"."user_preferences" (
-    "id" uuid DEFAULT uuid_generate_v4() NOT NULL,
-    "user_id" uuid NOT NULL,
-    "theme" varchar(20) DEFAULT 'light' NOT NULL,
-    "timezone" varchar(50) DEFAULT 'UTC',
-    "email_notifications" boolean DEFAULT true,
-    "push_notifications" boolean DEFAULT false,
-    "metadata" jsonb DEFAULT '{}'::jsonb,
-    "created_at" timestamp with time zone DEFAULT CURRENT_TIMESTAMP,
-    "updated_at" timestamp with time zone DEFAULT CURRENT_TIMESTAMP
-);
-
-ALTER TABLE ONLY "public"."user_preferences"
-    ADD CONSTRAINT "user_preferences_pkey" PRIMARY KEY ("id");
-
--- ... additional DDL statements
-```
-
-**DOWN Migration** (`*_down.sql`):
-
-```sql
--- Migration: 20240830_143022_add_user_preferences DOWN
--- Generated: 2024-08-30T14:30:22.123456
-
-DROP TABLE IF EXISTS "public"."user_preferences";
-ALTER TABLE "public"."boards" DROP COLUMN IF EXISTS "view_count";
-DROP INDEX IF EXISTS "public"."idx_boards_view_count";
-```
-
-### Step 3: Review and Test Migration
-
-Before applying, review the generated migration:
-
-```bash
-# Review the UP migration
-cat migrations/generated/*add_user_preferences_up.sql
-
-# Review the DOWN migration (for rollback)
-cat migrations/generated/*add_user_preferences_down.sql
-
-# Test on a copy of your database first (recommended)
-pg_dump boards_dev > backup.sql
-createdb boards_test
-psql boards_test < backup.sql
-psql boards_test < migrations/generated/*add_user_preferences_up.sql
-```
-
-### Step 4: Apply Migration to Database
-
-Apply the generated migration to your database:
-
-```bash
-# Method 1: Direct application with psql
-psql boards_dev < migrations/generated/*add_user_preferences_up.sql
-
-# Method 2: Using the migration runner (tracks applied migrations)
-python migrations/migration_runner.py up --target add_user_preferences
-
-# Verify the changes
-psql boards_dev -c "\dt user_preferences"
-psql boards_dev -c "SELECT column_name, data_type FROM information_schema.columns WHERE table_name = 'user_preferences';"
-```
-
-### Step 5: Regenerate SQLAlchemy Models
-
-Update the Python models to reflect the new schema:
-
-```bash
-# Generate models from current database schema
-python scripts/generate_models.py
-
-# This updates: src/boards/database/models.py
-```
-
-#### Verify Generated Models
-
-```bash
-# Check that new models are generated
-grep -A 10 "class UserPreferences" src/boards/database/models.py
-
-# Test model import
-python -c "from boards.database.models import UserPreferences; print(UserPreferences.__table__.columns.keys())"
-```
-
-### Step 6: Update GraphQL Schema (if needed)
-
-If you added new tables/columns, update the GraphQL types:
-
-```bash
-# Add new types to GraphQL schema
-vim src/boards/graphql/types/user_preferences.py
-
-# Add resolvers
-vim src/boards/graphql/resolvers/user_preferences.py
-
-# Update queries/mutations
-vim src/boards/graphql/queries/root.py
-```
-
-### Step 7: Test the Changes
-
-```bash
-# Start the development server
-uvicorn boards.api.app:app --reload --port 8000
-
-# Test GraphQL endpoint
-curl -X POST http://localhost:8088/graphql \
-  -H "Content-Type: application/json" \
-  -d '{"query": "{ __schema { types { name } } }"}'
-
-# Or open GraphiQL: http://localhost:8088/graphql
-```
-
-## Rollback Procedure
-
-If you need to undo a migration:
-
-### Method 1: Using DOWN Migration
-
-```bash
-# Apply the DOWN migration
-psql boards_dev < migrations/generated/*add_user_preferences_down.sql
-
-# Regenerate models to reflect rollback
-python scripts/generate_models.py
-```
-
-### Method 2: Using Migration Runner
-
-```bash
-# Rollback to previous migration
-python migrations/migration_runner.py down --target 001_initial_schema
-
-# Or rollback completely
-python migrations/migration_runner.py down --target zero
-```
-
-### Method 3: Database Restore
-
-```bash
-# If you have a backup
-dropdb boards_dev
-createdb boards_dev
-psql boards_dev < backup.sql
-```
-
-## Common Migration Patterns
-
-### Adding a New Table
-
-```sql
--- 003_add_analytics.sql
-CREATE TABLE page_views (
-    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
-    tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE,
-    user_id UUID REFERENCES users(id) ON DELETE SET NULL,
-    page_path VARCHAR(255) NOT NULL,
-    viewed_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-    metadata JSONB DEFAULT '{}'
-);
-
-CREATE INDEX idx_page_views_tenant ON page_views(tenant_id);
-CREATE INDEX idx_page_views_user ON page_views(user_id);
-CREATE INDEX idx_page_views_path ON page_views(page_path);
-CREATE INDEX idx_page_views_date ON page_views(viewed_at);
-```
-
-### Adding Columns to Existing Table
-
-```sql
--- 004_add_board_features.sql
--- Add new columns to boards table
-ALTER TABLE boards ADD COLUMN is_archived BOOLEAN DEFAULT FALSE;
-ALTER TABLE boards ADD COLUMN archived_at TIMESTAMP WITH TIME ZONE;
-ALTER TABLE boards ADD COLUMN tags TEXT[] DEFAULT '{}';
-
--- Add indexes
-CREATE INDEX idx_boards_archived ON boards(is_archived);
-CREATE INDEX idx_boards_archived_at ON boards(archived_at);
-CREATE INDEX idx_boards_tags ON boards USING GIN(tags);
-
--- Add constraint
-ALTER TABLE boards ADD CONSTRAINT check_archived_date
-    CHECK (is_archived = FALSE OR archived_at IS NOT NULL);
-```
-
-### Creating Indexes
-
-```sql
--- 005_optimize_queries.sql
--- Add indexes for common query patterns
-
--- Composite index for board member queries
-CREATE INDEX idx_board_members_board_role ON board_members(board_id, role);
-
--- Partial index for active generations
-CREATE INDEX idx_generations_active ON generations(board_id, created_at)
-    WHERE status IN ('pending', 'processing');
-
--- Text search index
-CREATE INDEX idx_boards_title_search ON boards USING gin(to_tsvector('english', title));
-```
-
-### Modifying Column Types
-
-```sql
--- 006_extend_varchar_limits.sql
--- Increase varchar limits based on usage patterns
-
-ALTER TABLE users ALTER COLUMN display_name TYPE VARCHAR(500);
-ALTER TABLE boards ALTER COLUMN title TYPE VARCHAR(500);
-ALTER TABLE generations ALTER COLUMN generator_name TYPE VARCHAR(200);
-```
-
-### Adding Foreign Keys
-
-```sql
--- 007_add_missing_constraints.sql
--- Add foreign key constraints that were missed
-
--- Add foreign key from provider_configs to something
-ALTER TABLE provider_configs
-    ADD CONSTRAINT fk_provider_configs_updated_by
-    FOREIGN KEY (updated_by) REFERENCES users(id);
-
--- Add check constraints
-ALTER TABLE credit_transactions
-    ADD CONSTRAINT check_positive_amount CHECK (amount >= 0);
-```
-
-## Production Deployment
-
-### Pre-deployment Checklist
-
-- [ ] Migration tested on development database
-- [ ] Migration tested on staging database with production-like data
-- [ ] Backup created before deployment
-- [ ] Rollback plan prepared
-- [ ] Downtime requirements communicated
-- [ ] Performance impact assessed
-
-### Deployment Steps
-
-```bash
-# 1. Create backup
-pg_dump boards_production > backup_$(date +%Y%m%d_%H%M%S).sql
-
-# 2. Apply migration
-psql boards_production < migrations/generated/*_migration_up.sql
-
-# 3. Regenerate models on server
-python scripts/generate_models.py
-
-# 4. Restart application
-systemctl restart boards-api
-
-# 5. Verify deployment
-curl -f http://localhost:8088/health
-```
-
-### Zero-downtime Migrations
-
-For large tables, consider these strategies:
-
-1. **Add columns as nullable first**:
-
-```sql
--- Step 1: Add nullable column
-ALTER TABLE large_table ADD COLUMN new_column VARCHAR(255);
-
--- Step 2: Backfill data (in application code)
--- Step 3: Add NOT NULL constraint
-ALTER TABLE large_table ALTER COLUMN new_column SET NOT NULL;
-```
-
-2. **Use CREATE INDEX CONCURRENTLY**:
-
-```sql
--- Non-blocking index creation
-CREATE INDEX CONCURRENTLY idx_large_table_new_column ON large_table(new_column);
-```
-
-3. **Staged rollouts**: Apply changes gradually across multiple deployments
-
-## Troubleshooting
-
-### Common Issues
-
-**Migration fails with constraint violation:**
-
-```bash
-# Check existing data
-psql boards_dev -c "SELECT COUNT(*) FROM users WHERE email IS NULL;"
-
-# Fix data first, then retry migration
-psql boards_dev -c "UPDATE users SET email = 'unknown@example.com' WHERE email IS NULL;"
-```
-
-**Generated models are incorrect:**
-
-```bash
-# Clear and regenerate
-rm src/boards/database/models.py
-python scripts/generate_models.py
-```
-
-**Migration generates no changes:**
-
-```bash
-# Check if schema files match current database
-python scripts/generate_migration.py --name test --dry-run
-```
-
-**Schema conflicts:**
-
-```bash
-# Compare schemas manually
-pg_dump boards_dev --schema-only > current_schema.sql
-# Apply DDL files to temp database and compare
-```
-
-### Getting Help
-
-1. Check the generated migration files first
-2. Test on a development database copy
-3. Review PostgreSQL logs: `tail -f /var/log/postgresql/postgresql.log`
-4. Use `EXPLAIN ANALYZE` for performance testing
-
-## Best Practices Summary
-
-✅ **DO:**
-
-- Always test migrations on development/staging first
-- Use descriptive migration names
-- Include proper indexes and constraints
-- Document complex changes
-- Create backups before production migrations
-- Use transactions for complex migrations
-
-❌ **DON'T:**
-
-- Apply migrations directly to production without testing
-- Create migrations that can't be rolled back
-- Forget to regenerate models after schema changes
-- Skip index creation for new columns
-- Make breaking changes without a migration strategy
-
-## File Structure Reference
-
-```
-packages/backend/
-├── migrations/
-│   ├── schemas/              # DDL source files (edit these)
-│   │   ├── 001_initial_schema.sql
-│   │   ├── 002_add_user_preferences.sql
-│   │   └── 003_add_analytics.sql
-│   ├── generated/            # Generated migration scripts (don't edit)
-│   │   ├── 20240830_143022_add_user_preferences_up.sql
-│   │   ├── 20240830_143022_add_user_preferences_down.sql
-│   │   └── ...
-│   └── migration_runner.py   # Migration application tool
-├── scripts/
-│   ├── generate_models.py    # DDL → SQLAlchemy models
-│   └── generate_migration.py # Schema diff → migration scripts
-└── src/boards/database/
-    └── models.py             # Generated SQLAlchemy models (don't edit)
-```
-
-Remember: The schema files in `migrations/schemas/` are your source of truth. Everything else is generated!