@caoscompanybr/merlin 3.5.0

package/.merlin-core/skills/general/database/SKILL.md

@@ -0,0 +1,561 @@
+---
+name: database
+description: Database expertise covering SQL/NoSQL patterns, ORMs, migrations, indexing, query optimization, and PostgreSQL deep dives (RLS, locking, connection pooling, monitoring)
+license: Apache-2.0
+metadata:
+  author: merlin-framework
+  version: "3.0.0"
+  auto_activate: [database, sql-query, postgres, mysql, mongodb, prisma-schema]
+---
+
+# Database Design & Optimization
+
+Expert knowledge of relational (SQL) and non-relational (NoSQL) databases, ORM patterns, schema design, migrations, indexing strategies, and query optimization.
+
+## Schema Design
+
+### Relational Database Best Practices
+
+```sql
+-- Users table with proper constraints
+CREATE TABLE users (
+  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+  email VARCHAR(255) UNIQUE NOT NULL,
+  username VARCHAR(50) UNIQUE NOT NULL,
+  password_hash VARCHAR(255) NOT NULL,
+  created_at TIMESTAMP NOT NULL DEFAULT NOW(),
+  updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
+  deleted_at TIMESTAMP NULL,  -- Soft delete
+
+  CONSTRAINT email_format CHECK (email ~* '^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$')
+);
+
+-- Posts table with foreign key
+CREATE TABLE posts (
+  id BIGSERIAL PRIMARY KEY,
+  user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+  title VARCHAR(200) NOT NULL,
+  content TEXT,
+  status VARCHAR(20) NOT NULL DEFAULT 'draft',
+  published_at TIMESTAMP NULL,
+  created_at TIMESTAMP NOT NULL DEFAULT NOW(),
+  updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
+
+  CONSTRAINT status_values CHECK (status IN ('draft', 'published', 'archived'))
+);
+
+-- Many-to-many relationship
+CREATE TABLE post_tags (
+  post_id BIGINT NOT NULL REFERENCES posts(id) ON DELETE CASCADE,
+  tag_id INTEGER NOT NULL REFERENCES tags(id) ON DELETE CASCADE,
+  created_at TIMESTAMP NOT NULL DEFAULT NOW(),
+
+  PRIMARY KEY (post_id, tag_id)
+);
+
+-- Note: gen_random_uuid() creates UUIDv4 (random). For large tables,
+-- prefer UUIDv7 (time-ordered) to avoid index fragmentation:
+-- id UUID PRIMARY KEY DEFAULT uuid_generate_v7()  -- requires pg_uuidv7
+
+-- Indexes for common queries
+CREATE INDEX idx_posts_user_id ON posts(user_id);
+CREATE INDEX idx_posts_status ON posts(status) WHERE status = 'published';
+CREATE INDEX idx_posts_published_at ON posts(published_at DESC) WHERE published_at IS NOT NULL;
+```
+
+**Primary key strategy:** `bigint GENERATED ALWAYS AS IDENTITY` for single-database. UUIDv7 for distributed systems. Avoid UUIDv4 on large tables (random inserts = index fragmentation).
+
+### Normalization
+
+```
+1NF: Atomic values, no repeating groups
+2NF: 1NF + No partial dependencies
+3NF: 2NF + No transitive dependencies
+
+-- BAD: Denormalized (repeated data)
+users: id | name | email | address | city | state | zip
+
+-- GOOD: Normalized
+users: id | name | email | address_id
+addresses: id | street | city | state | zip
+```
+
+### Denormalization (When appropriate)
+
+```sql
+-- Add computed/cached fields for performance
+ALTER TABLE users ADD COLUMN post_count INTEGER DEFAULT 0;
+
+-- Update trigger to maintain count
+CREATE TRIGGER update_post_count
+AFTER INSERT ON posts
+FOR EACH ROW
+EXECUTE FUNCTION increment_user_post_count();
+
+-- Use for read-heavy workloads where consistency can be eventual
+```
+
+## Index Strategies
+
+### Index Types
+
+```sql
+-- B-tree index (default, good for equality and range)
+CREATE INDEX idx_users_email ON users(email);
+
+-- Unique index
+CREATE UNIQUE INDEX idx_users_username ON users(username);
+
+-- Composite index (order matters!)
+CREATE INDEX idx_posts_user_status ON posts(user_id, status);
+
+-- Partial index (smaller, faster for specific queries)
+CREATE INDEX idx_active_users ON users(email) WHERE deleted_at IS NULL;
+
+-- Expression index
+CREATE INDEX idx_users_lower_email ON users(LOWER(email));
+
+-- Full-text search (PostgreSQL)
+CREATE INDEX idx_posts_content_fts ON posts USING GIN(to_tsvector('english', content));
+
+-- JSONB index (PostgreSQL)
+CREATE INDEX idx_metadata ON users USING GIN(metadata);
+```
+
+### Index Type Selection
+
+| Type   | Best For                                            | Syntax                             |
+| ------ | --------------------------------------------------- | ---------------------------------- |
+| B-tree | `=`, `<`, `>`, `BETWEEN`, `ORDER BY` (default)      | `CREATE INDEX ... ON t(col)`       |
+| GIN    | Arrays, JSONB, full-text search                     | `CREATE INDEX ... USING GIN(col)`  |
+| GiST   | Geometry, ranges, nearest-neighbor                  | `CREATE INDEX ... USING GiST(col)` |
+| BRIN   | Large time-series (100x smaller than B-tree)        | `CREATE INDEX ... USING BRIN(col)` |
+| Hash   | Equality-only (slightly faster than B-tree for `=`) | `CREATE INDEX ... USING HASH(col)` |
+
+```sql
+-- Covering index (index-only scan, avoids heap fetch — 2-5x faster)
+CREATE INDEX idx_users_email_covering ON users (email) INCLUDE (name, created_at);
+```
+
+### Index Best Practices
+
+```sql
+-- Good: Index on foreign keys
+CREATE INDEX idx_posts_user_id ON posts(user_id);
+
+-- Good: Index on frequently filtered columns
+CREATE INDEX idx_posts_status ON posts(status);
+
+-- Good: Composite index for common multi-column queries
+-- Query: WHERE user_id = ? AND status = ? ORDER BY created_at DESC
+CREATE INDEX idx_posts_user_status_created ON posts(user_id, status, created_at DESC);
+
+-- Bad: Over-indexing (slows writes)
+-- Don't index every column
+
+-- Bad: Redundant indexes
+CREATE INDEX idx_posts_user_id ON posts(user_id);
+CREATE INDEX idx_posts_user_status ON posts(user_id, status);
+-- First index is redundant (covered by second)
+```
+
+### Checking Index Usage
+
+```sql
+-- PostgreSQL: Check unused indexes
+SELECT schemaname, tablename, indexname, idx_scan
+FROM pg_stat_user_indexes
+WHERE idx_scan = 0
+ORDER BY schemaname, tablename;
+
+-- Explain query plan
+EXPLAIN ANALYZE
+SELECT * FROM posts WHERE user_id = '123' AND status = 'published';
+```
+
+### Find Missing Foreign Key Indexes
+
+```sql
+SELECT conrelid::regclass AS table_name, a.attname AS fk_column
+FROM pg_constraint c
+JOIN pg_attribute a ON a.attrelid = c.conrelid AND a.attnum = ANY(c.conkey)
+WHERE c.contype = 'f'
+  AND NOT EXISTS (
+    SELECT 1 FROM pg_index i
+    WHERE i.indrelid = c.conrelid AND a.attnum = ANY(i.indkey)
+  );
+```
+
+## N+1 Query Prevention
+
+### The Problem
+
+```typescript
+// BAD: N+1 queries
+const users = await User.findAll(); // 1 query
+for (const user of users) {
+  const posts = await Post.findAll({ where: { userId: user.id } }); // N queries
+}
+// Total: 1 + N queries
+```
+
+### Solutions
+
+**Eager Loading (Sequelize)**
+
+```typescript
+// GOOD: 1 or 2 queries total
+const users = await User.findAll({
+  include: [{ model: Post }],
+});
+// users[0].posts is already loaded
+```
+
+**Eager Loading (Prisma)**
+
+```typescript
+const users = await prisma.user.findMany({
+  include: {
+    posts: true,
+    profile: true,
+  },
+});
+```
+
+**DataLoader (GraphQL)**
+
+```typescript
+const userLoader = new DataLoader(async (userIds) => {
+  const users = await User.findAll({ where: { id: userIds } });
+  return userIds.map((id) => users.find((u) => u.id === id));
+});
+
+// Batches and caches requests
+const user = await userLoader.load(userId);
+```
+
+**Raw SQL with JOIN**
+
+```sql
+-- Single query with JOIN
+SELECT u.*, p.*
+FROM users u
+LEFT JOIN posts p ON p.user_id = u.id
+WHERE u.id = ANY($1);
+```
+
+## Query Optimization
+
+### Use EXPLAIN ANALYZE
+
+```sql
+EXPLAIN ANALYZE
+SELECT u.name, COUNT(p.id) as post_count
+FROM users u
+LEFT JOIN posts p ON p.user_id = u.id
+WHERE u.created_at > '2026-01-01'
+GROUP BY u.id, u.name
+ORDER BY post_count DESC
+LIMIT 10;
+
+-- Look for:
+-- - Seq Scan (bad for large tables, add index)
+-- - Index Scan (good)
+-- - Nested Loop (can be slow)
+-- - Hash Join (good for large datasets)
+```
+
+### Optimize Common Patterns
+
+```sql
+-- BAD: SELECT *
+SELECT * FROM users;
+
+-- GOOD: Select only needed columns
+SELECT id, name, email FROM users;
+
+-- BAD: OR conditions (can't use index efficiently)
+SELECT * FROM users WHERE email = 'a@b.com' OR username = 'user';
+
+-- GOOD: Use UNION for OR on different columns
+SELECT * FROM users WHERE email = 'a@b.com'
+UNION
+SELECT * FROM users WHERE username = 'user';
+
+-- BAD: Function on indexed column (can't use index)
+SELECT * FROM users WHERE LOWER(email) = 'test@example.com';
+
+-- GOOD: Store lowercase, or use expression index
+SELECT * FROM users WHERE email = 'test@example.com';
+
+-- GOOD: Limit results
+SELECT * FROM posts ORDER BY created_at DESC LIMIT 100;
+
+-- GOOD: Use EXISTS instead of IN for subqueries
+SELECT * FROM users u
+WHERE EXISTS (SELECT 1 FROM posts p WHERE p.user_id = u.id);
+```
+
+## Migrations
+
+### Migration Best Practices
+
+```sql
+-- Use transactions (automatic in most tools)
+BEGIN;
+
+-- Add column with default (can lock table)
+ALTER TABLE users ADD COLUMN role VARCHAR(20) DEFAULT 'user' NOT NULL;
+
+-- For large tables, add nullable first, then backfill
+ALTER TABLE users ADD COLUMN role VARCHAR(20);
+-- Backfill in batches
+UPDATE users SET role = 'user' WHERE role IS NULL LIMIT 1000;
+-- Then add constraint
+ALTER TABLE users ALTER COLUMN role SET NOT NULL;
+
+-- Add index concurrently (PostgreSQL, doesn't lock)
+CREATE INDEX CONCURRENTLY idx_users_role ON users(role);
+
+-- Rename column
+ALTER TABLE users RENAME COLUMN username TO user_name;
+
+-- Drop column (can't be easily rolled back)
+ALTER TABLE users DROP COLUMN old_field;
+
+COMMIT;
+```
+
+### Migration Safety
+
+```typescript
+// Good migration structure (Prisma)
+// migrations/001_add_user_role.sql
+
+-- Migration
+ALTER TABLE users ADD COLUMN role VARCHAR(20);
+
+-- Rollback (separate file or comment)
+-- ALTER TABLE users DROP COLUMN role;
+
+// Version control all migrations
+// Never modify existing migrations in production
+// Always test rollback before deploying
+```
+
+## ORM Patterns
+
+### Prisma
+
+```typescript
+// Schema definition
+model User {
+  id        String   @id @default(uuid())
+  email     String   @unique
+  posts     Post[]
+  profile   Profile?
+  createdAt DateTime @default(now())
+  updatedAt DateTime @updatedAt
+
+  @@index([email])
+}
+
+model Post {
+  id        Int      @id @default(autoincrement())
+  title     String
+  content   String?
+  author    User     @relation(fields: [authorId], references: [id])
+  authorId  String
+  createdAt DateTime @default(now())
+
+  @@index([authorId])
+}
+
+// Queries
+const users = await prisma.user.findMany({
+  where: { email: { contains: '@example.com' } },
+  include: { posts: true },
+  orderBy: { createdAt: 'desc' },
+  take: 10,
+});
+
+// Transaction
+await prisma.$transaction(async (tx) => {
+  const user = await tx.user.create({ data: userData });
+  await tx.profile.create({ data: { userId: user.id, ...profileData } });
+});
+```
+
+### Sequelize
+
+```typescript
+// Model definition
+class User extends Model {
+  declare id: string;
+  declare email: string;
+  declare posts?: Post[];
+}
+
+User.init(
+  {
+    id: {
+      type: DataTypes.UUID,
+      defaultValue: DataTypes.UUIDV4,
+      primaryKey: true,
+    },
+    email: { type: DataTypes.STRING, allowNull: false, unique: true },
+  },
+  { sequelize },
+);
+
+// Associations
+User.hasMany(Post, { foreignKey: "authorId" });
+Post.belongsTo(User, { foreignKey: "authorId" });
+
+// Queries
+const users = await User.findAll({
+  where: { email: { [Op.like]: "%@example.com" } },
+  include: [Post],
+  order: [["createdAt", "DESC"]],
+  limit: 10,
+});
+```
+
+### TypeORM
+
+```typescript
+@Entity()
+class User {
+  @PrimaryGeneratedColumn("uuid")
+  id: string;
+
+  @Column({ unique: true })
+  email: string;
+
+  @OneToMany(() => Post, (post) => post.author)
+  posts: Post[];
+
+  @CreateDateColumn()
+  createdAt: Date;
+}
+
+// Queries
+const users = await userRepository.find({
+  where: { email: Like("%@example.com") },
+  relations: ["posts"],
+  order: { createdAt: "DESC" },
+  take: 10,
+});
+```
+
+## NoSQL Patterns (MongoDB)
+
+### Schema Design
+
+```typescript
+// Embedded documents (1-to-few)
+{
+  _id: ObjectId("..."),
+  name: "John Doe",
+  email: "john@example.com",
+  addresses: [
+    { street: "123 Main St", city: "NYC", type: "home" },
+    { street: "456 Work Ave", city: "NYC", type: "work" }
+  ]
+}
+
+// Referenced documents (1-to-many)
+// User document
+{ _id: ObjectId("user1"), name: "John Doe" }
+
+// Post documents (separate collection)
+{ _id: ObjectId("post1"), userId: ObjectId("user1"), title: "Post 1" }
+{ _id: ObjectId("post2"), userId: ObjectId("user1"), title: "Post 2" }
+```
+
+### Indexing
+
+```javascript
+// Create indexes
+db.users.createIndex({ email: 1 }, { unique: true });
+db.posts.createIndex({ userId: 1, createdAt: -1 });
+db.posts.createIndex({ title: "text", content: "text" }); // Text search
+
+// Compound index
+db.posts.createIndex({ status: 1, publishedAt: -1 });
+
+// TTL index (auto-delete after time)
+db.sessions.createIndex({ createdAt: 1 }, { expireAfterSeconds: 3600 });
+```
+
+### Query Patterns
+
+```javascript
+// Find with filtering and projection
+db.users
+  .find({ status: "active", age: { $gte: 18 } }, { name: 1, email: 1, _id: 0 })
+  .sort({ createdAt: -1 })
+  .limit(10);
+
+// Aggregation pipeline
+db.posts.aggregate([
+  { $match: { status: "published" } },
+  { $group: { _id: "$userId", count: { $sum: 1 } } },
+  { $sort: { count: -1 } },
+  { $limit: 10 },
+]);
+
+// Lookup (join)
+db.posts.aggregate([
+  {
+    $lookup: {
+      from: "users",
+      localField: "userId",
+      foreignField: "_id",
+      as: "author",
+    },
+  },
+  { $unwind: "$author" },
+]);
+```
+
+## PostgreSQL Deep Dives
+
+For Postgres-specific operational depth beyond the design patterns above:
+
+- **Row-Level Security** — Tenant isolation, policy performance (100x speedup patterns), privilege management. Read `references/postgres-rls.md`.
+- **Concurrency & Locking** — Advisory locks, deadlock prevention, short transactions, SKIP LOCKED queue pattern. Read `references/postgres-concurrency.md`.
+- **Connection Management** — Pool sizing formula, transaction vs session mode, max_connections calculation. Read `references/postgres-connections.md`.
+- **Monitoring & Diagnostics** — EXPLAIN ANALYZE red flags, pg_stat_statements, vacuum/analyze tuning. Read `references/postgres-monitoring.md`.
+- **Data Access Patterns** — Batch inserts, cursor pagination (vs OFFSET), upsert (ON CONFLICT), table partitioning. Read `references/postgres-data-patterns.md`.
+
+## Common Pitfalls
+
+1. **No indexes on foreign keys**
+2. **Missing WHERE clause on soft deletes**
+3. **N+1 queries (use eager loading)**
+4. **Using SELECT \* instead of specific columns**
+5. **Not using transactions for multi-step operations**
+6. **Storing passwords in plain text**
+7. **No connection pooling**
+8. **Ignoring query performance (no EXPLAIN)**
+9. **Over-indexing (every column)**
+10. **Not planning for data growth**
+
+## Best Practices
+
+1. Always index foreign keys
+2. Use transactions for data consistency
+3. Implement soft deletes with deleted_at
+4. Use UUIDs for distributed systems
+5. Add created_at and updated_at timestamps
+6. Use connection pooling
+7. Monitor slow queries
+8. Regular VACUUM (PostgreSQL) or OPTIMIZE (MySQL)
+9. Backup regularly with tested restore process
+10. Use prepared statements to prevent SQL injection
+11. Implement proper error handling
+12. Use migrations for all schema changes
+13. Test migrations and rollbacks
+14. Monitor database performance metrics
+15. Plan for horizontal/vertical scaling