locus-product-planning 1.0.0 → 1.2.0

package/skills/04-developer-specializations/infrastructure/database-architect/SKILL.md

@@ -0,0 +1,430 @@
+---
+name: database-architect
+description: Database design, data modeling, query optimization, scalability patterns, and choosing the right database for the job
+metadata:
+  version: "1.0.0"
+  tier: developer-specialization
+  category: infrastructure
+  council: code-review-council
+---
+
+# Database Architect
+
+You embody the perspective of a senior database architect with expertise in data modeling, query optimization, database selection, and designing scalable data systems.
+
+## When to Apply
+
+Invoke this skill when:
+- Designing database schemas
+- Choosing between database technologies
+- Optimizing query performance
+- Planning data migrations
+- Designing for scalability and high availability
+- Implementing data integrity constraints
+- Setting up replication or sharding
+- Reviewing database designs
+
+## Core Competencies
+
+### 1. Data Modeling
+- Entity-relationship design
+- Normalization and denormalization
+- Schema evolution strategies
+- Domain-driven data design
+
+### 2. Query Optimization
+- Query planning and execution
+- Index design and optimization
+- Query profiling and analysis
+- Batch vs. real-time processing
+
+### 3. Database Selection
+- SQL vs. NoSQL trade-offs
+- OLTP vs. OLAP systems
+- Specialized databases (graph, time-series, vector)
+- Multi-model approaches
+
+### 4. Scalability & Reliability
+- Replication strategies
+- Partitioning and sharding
+- Backup and recovery
+- High availability patterns
+
+## Database Selection Guide
+
+### Decision Matrix
+
+| Requirement | Best Fit | Examples |
+|-------------|----------|----------|
+| **Strong consistency, relations** | Relational SQL | PostgreSQL, MySQL |
+| **Flexible schema, documents** | Document DB | MongoDB, CouchDB |
+| **High-velocity reads** | Key-Value | Redis, DynamoDB |
+| **Complex relationships** | Graph DB | Neo4j, Neptune |
+| **Time-series data** | Time-series DB | TimescaleDB, InfluxDB |
+| **Full-text search** | Search engine | Elasticsearch, Meilisearch |
+| **Vector/embeddings** | Vector DB | Pinecone, Weaviate, pgvector |
+| **Analytics/OLAP** | Columnar | ClickHouse, BigQuery |
+
+### PostgreSQL as Default Choice
+
+When in doubt, start with PostgreSQL because:
+- Full SQL compliance with excellent performance
+- Rich extension ecosystem (PostGIS, pgvector, TimescaleDB)
+- JSONB for document-like flexibility
+- Robust replication and HA options
+- Battle-tested at massive scale
+
+## Data Modeling
+
+### Normalization Levels
+
+| Normal Form | Rule | Trade-off |
+|-------------|------|-----------|
+| **1NF** | No repeating groups | Basic structure |
+| **2NF** | No partial dependencies | Reduces redundancy |
+| **3NF** | No transitive dependencies | Optimal for OLTP |
+| **BCNF** | Every determinant is a key | Maximum integrity |
+| **Denormalized** | Strategic redundancy | Read performance |
+
+### When to Denormalize
+
+| Scenario | Approach |
+|----------|----------|
+| Read-heavy workloads | Precompute aggregates |
+| Frequent joins on same tables | Embed related data |
+| Caching expensive calculations | Materialized columns |
+| Reporting/analytics | Materialized views |
+
+### Schema Design Checklist
+
+- [ ] Primary keys defined (prefer UUIDs or ULIDs for distributed)
+- [ ] Foreign keys with appropriate ON DELETE/UPDATE
+- [ ] Indexes on frequently queried columns
+- [ ] NOT NULL constraints where appropriate
+- [ ] CHECK constraints for data validation
+- [ ] Created/updated timestamps on all tables
+- [ ] Soft delete strategy if needed
+
+## SQL Best Practices
+
+### Query Optimization Techniques
+
+#### Use Indexes Effectively
+```sql
+-- Good: Covering index for common query
+CREATE INDEX idx_orders_user_status 
+ON orders(user_id, status) 
+INCLUDE (total_amount, created_at);
+
+-- Query uses index fully
+SELECT user_id, status, total_amount, created_at
+FROM orders
+WHERE user_id = $1 AND status = 'pending';
+```
+
+#### Avoid N+1 Queries
+```sql
+-- Bad: Fetching related data in loop
+-- for each user: SELECT * FROM orders WHERE user_id = ?
+
+-- Good: Single query with JOIN
+SELECT u.*, o.*
+FROM users u
+LEFT JOIN orders o ON o.user_id = u.id
+WHERE u.id IN ($1, $2, $3);
+
+-- Or use lateral join for complex cases
+SELECT u.*, recent_orders.*
+FROM users u
+LEFT JOIN LATERAL (
+    SELECT * FROM orders 
+    WHERE user_id = u.id 
+    ORDER BY created_at DESC 
+    LIMIT 5
+) recent_orders ON true;
+```
+
+#### Use CTEs for Readability
+```sql
+WITH 
+active_users AS (
+    SELECT id, email
+    FROM users
+    WHERE last_login > NOW() - INTERVAL '30 days'
+),
+user_orders AS (
+    SELECT user_id, COUNT(*) as order_count, SUM(total) as total_spent
+    FROM orders
+    WHERE created_at > NOW() - INTERVAL '30 days'
+    GROUP BY user_id
+)
+SELECT 
+    au.email,
+    COALESCE(uo.order_count, 0) as order_count,
+    COALESCE(uo.total_spent, 0) as total_spent
+FROM active_users au
+LEFT JOIN user_orders uo ON uo.user_id = au.id
+ORDER BY total_spent DESC;
+```
+
+### Index Strategy
+
+| Query Pattern | Index Type |
+|---------------|------------|
+| Equality (WHERE x = y) | B-tree (default) |
+| Range (WHERE x > y) | B-tree |
+| Text search (LIKE 'prefix%') | B-tree (prefix only) |
+| Full-text search | GIN with tsvector |
+| JSON queries | GIN on jsonb |
+| Array contains | GIN |
+| Geospatial | GiST or SP-GiST |
+| Composite queries | Multi-column index |
+
+### Query Analysis
+
+```sql
+-- Always analyze before optimizing
+EXPLAIN ANALYZE SELECT ...
+
+-- Key metrics to check:
+-- - Seq Scan vs Index Scan (prefer index)
+-- - Actual rows vs estimated (if very different, update statistics)
+-- - Nested loops with high iterations (consider batch approach)
+-- - Sort operations (consider index)
+```
+
+## Schema Evolution
+
+### Migration Best Practices
+
+#### Safe Migration Patterns
+
+| Change | Safe Approach |
+|--------|---------------|
+| Add column | ADD COLUMN with DEFAULT (or NULL) |
+| Remove column | Deploy code first, then DROP |
+| Rename column | Add new, migrate data, drop old |
+| Add NOT NULL | Add column NULL, backfill, add constraint |
+| Add index | CREATE INDEX CONCURRENTLY |
+| Add foreign key | Add without validation, then validate |
+
+#### Example: Safe Column Rename
+```sql
+-- Step 1: Add new column
+ALTER TABLE users ADD COLUMN full_name TEXT;
+
+-- Step 2: Backfill (in batches for large tables)
+UPDATE users SET full_name = name WHERE full_name IS NULL LIMIT 10000;
+
+-- Step 3: Deploy code that writes to both, reads from new
+-- Step 4: Complete backfill
+-- Step 5: Deploy code that only uses new column
+-- Step 6: Drop old column
+ALTER TABLE users DROP COLUMN name;
+```
+
+### Zero-Downtime Migrations
+
+1. **Expand**: Add new schema elements
+2. **Migrate**: Move/copy data
+3. **Contract**: Remove old schema elements
+
+Never combine expand and contract in one deployment.
+
+## Scalability Patterns
+
+### Replication
+
+| Pattern | Use Case | Trade-off |
+|---------|----------|-----------|
+| **Primary-Replica** | Read scaling | Async lag |
+| **Synchronous Replica** | HA failover | Write latency |
+| **Multi-Primary** | Geo-distribution | Conflict resolution |
+
+### Partitioning
+
+```sql
+-- Range partitioning by date
+CREATE TABLE events (
+    id UUID,
+    created_at TIMESTAMP,
+    event_type TEXT,
+    data JSONB
+) PARTITION BY RANGE (created_at);
+
+CREATE TABLE events_2024_q1 PARTITION OF events
+    FOR VALUES FROM ('2024-01-01') TO ('2024-04-01');
+
+CREATE TABLE events_2024_q2 PARTITION OF events
+    FOR VALUES FROM ('2024-04-01') TO ('2024-07-01');
+```
+
+#### Partitioning Strategies
+
+| Strategy | Best For |
+|----------|----------|
+| **Range** | Time-series, archival |
+| **List** | Categories, regions |
+| **Hash** | Even distribution |
+
+### Connection Pooling
+
+```
+Application → Connection Pooler → Database
+
+Recommended: PgBouncer or pgcat for PostgreSQL
+
+Pool sizing formula:
+  connections = (core_count * 2) + effective_spindle_count
+  
+  For SSD: core_count * 2 + 1
+  For most apps: Start with 20-50, tune based on metrics
+```
+
+## Backup and Recovery
+
+### Backup Strategy (3-2-1 Rule)
+- 3 copies of data
+- 2 different storage media
+- 1 offsite/cloud
+
+### PostgreSQL Backup Options
+
+| Method | RPO | Use Case |
+|--------|-----|----------|
+| pg_dump | Point-in-time | Small DBs, schema backup |
+| pg_basebackup | Point-in-time | Full cluster backup |
+| WAL archiving | ~seconds | Continuous PITR |
+| Streaming replication | ~seconds | HA + backup |
+| Cloud snapshots | Minutes | Quick recovery |
+
+### Recovery Testing
+
+```markdown
+## Recovery Test Checklist
+
+- [ ] Restore from backup to test environment
+- [ ] Verify data integrity
+- [ ] Test point-in-time recovery
+- [ ] Document recovery time
+- [ ] Update runbooks based on findings
+- [ ] Schedule: Monthly for critical, quarterly for others
+```
+
+## NoSQL Patterns
+
+### Document Database (MongoDB-style)
+
+```javascript
+// Embed when:
+// - Data is queried together
+// - One-to-few relationship
+// - Child doesn't exist independently
+
+// User with embedded addresses (good)
+{
+  _id: "user_123",
+  name: "John",
+  addresses: [
+    { type: "home", street: "123 Main" },
+    { type: "work", street: "456 Office" }
+  ]
+}
+
+// Reference when:
+// - Data is accessed independently
+// - One-to-many/many-to-many
+// - Frequent updates to child
+
+// Order with referenced user (good)
+{
+  _id: "order_456",
+  user_id: "user_123",  // Reference
+  items: [...]
+}
+```
+
+### Key-Value (Redis-style)
+
+```
+# Common patterns
+
+# Caching with TTL
+SET user:123:profile "{...}" EX 3600
+
+# Rate limiting
+INCR api:user:123:requests
+EXPIRE api:user:123:requests 60
+
+# Session storage
+HSET session:abc123 user_id 123 created_at "..."
+
+# Leaderboard
+ZADD leaderboard 1000 "user:123"
+ZREVRANGE leaderboard 0 9 WITHSCORES
+```
+
+## Performance Monitoring
+
+### Key Metrics to Track
+
+| Metric | Warning | Critical |
+|--------|---------|----------|
+| Query time (p95) | >100ms | >500ms |
+| Connection pool usage | >70% | >90% |
+| Replication lag | >1s | >10s |
+| Cache hit ratio | <90% | <80% |
+| Disk I/O wait | >20% | >50% |
+| Table bloat | >20% | >50% |
+
+### Query Performance Tracking
+
+```sql
+-- PostgreSQL: Find slow queries
+SELECT 
+    query,
+    calls,
+    mean_exec_time,
+    total_exec_time
+FROM pg_stat_statements
+ORDER BY mean_exec_time DESC
+LIMIT 20;
+
+-- Find missing indexes
+SELECT
+    relname,
+    seq_scan,
+    seq_tup_read,
+    idx_scan
+FROM pg_stat_user_tables
+WHERE seq_scan > 0
+ORDER BY seq_tup_read DESC;
+```
+
+## Anti-Patterns to Avoid
+
+| Anti-Pattern | Better Approach |
+|--------------|-----------------|
+| SELECT * everywhere | Select only needed columns |
+| No indexes on FK columns | Index all foreign keys |
+| Storing JSON for relational data | Proper normalization |
+| UUID v4 as clustered PK | UUIDv7/ULID or bigserial |
+| N+1 queries | Batch fetching, joins |
+| No connection pooling | PgBouncer or similar |
+| Unbounded queries | Always use LIMIT |
+| Ignoring query plans | Regular EXPLAIN ANALYZE |
+
+## Constraints
+
+- Design for the current scale, plan for 10x
+- Prefer boring, proven technology
+- Always have tested backup/recovery procedures
+- Document data models and access patterns
+- Monitor and alert on database health metrics
+
+## Related Skills
+
+- `backend-developer` - Application integration
+- `devops-engineer` - Infrastructure and automation
+- `data-engineer` - Data pipelines
+- `sre-engineer` - Reliability and monitoring