aigroup-workflow 2.1.2 → 2.2.1

package/skills/database/database-optimizer/SKILL.md

@@ -0,0 +1,147 @@
+---
+name: database-optimizer
+description: Optimizes database queries and improves performance across PostgreSQL and MySQL systems. Use when investigating slow queries, analyzing execution plans, or optimizing database performance. Invoke for index design, query rewrites, configuration tuning, partitioning strategies, lock contention resolution.
+license: MIT
+metadata:
+  author: https://github.com/Jeffallan
+  version: "1.1.1"
+  domain: infrastructure
+  triggers: database optimization, slow query, query performance, database tuning, index optimization, execution plan, EXPLAIN ANALYZE, database performance, PostgreSQL optimization, MySQL optimization
+  role: specialist
+  scope: optimization
+  output-format: analysis-and-code
+  related-skills: devops-engineer, postgres-pro, graphql-architect
+---
+
+# Database Optimizer
+
+Senior database optimizer with expertise in performance tuning, query optimization, and scalability across multiple database systems.
+
+## When to Use This Skill
+
+- Analyzing slow queries and execution plans
+- Designing optimal index strategies
+- Tuning database configuration parameters
+- Optimizing schema design and partitioning
+- Reducing lock contention and deadlocks
+- Improving cache hit rates and memory usage
+
+## Core Workflow
+
+1. **Analyze Performance** — Capture baseline metrics and run `EXPLAIN ANALYZE` before any changes
+2. **Identify Bottlenecks** — Find inefficient queries, missing indexes, config issues
+3. **Design Solutions** — Create index strategies, query rewrites, schema improvements
+4. **Implement Changes** — Apply optimizations incrementally with monitoring; validate each change before proceeding to the next
+5. **Validate Results** — Re-run `EXPLAIN ANALYZE`, compare costs, measure wall-clock improvement, document changes
+
+> ⚠️ Always test changes in non-production first. Revert immediately if write performance degrades or replication lag increases.
+
+## Reference Guide
+
+Load detailed guidance based on context:
+
+| Topic | Reference | Load When |
+|-------|-----------|-----------|
+| Query Optimization | `references/query-optimization.md` | Analyzing slow queries, execution plans |
+| Index Strategies | `references/index-strategies.md` | Designing indexes, covering indexes |
+| PostgreSQL Tuning | `references/postgresql-tuning.md` | PostgreSQL-specific optimizations |
+| MySQL Tuning | `references/mysql-tuning.md` | MySQL-specific optimizations |
+| Monitoring & Analysis | `references/monitoring-analysis.md` | Performance metrics, diagnostics |
+
+## Common Operations & Examples
+
+### Identify Top Slow Queries (PostgreSQL)
+```sql
+-- Requires pg_stat_statements extension
+SELECT query,
+       calls,
+       round(total_exec_time::numeric, 2)  AS total_ms,
+       round(mean_exec_time::numeric, 2)   AS mean_ms,
+       round(stddev_exec_time::numeric, 2) AS stddev_ms,
+       rows
+FROM   pg_stat_statements
+ORDER  BY mean_exec_time DESC
+LIMIT  20;
+```
+
+### Capture an Execution Plan
+```sql
+-- Use BUFFERS to expose cache hit vs. disk read ratio
+EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
+SELECT o.id, c.name
+FROM   orders o
+JOIN   customers c ON c.id = o.customer_id
+WHERE  o.status = 'pending'
+  AND  o.created_at > now() - interval '7 days';
+```
+
+### Reading EXPLAIN Output — Key Patterns to Find
+
+| Pattern | Symptom | Typical Remedy |
+|---------|---------|----------------|
+| `Seq Scan` on large table | High row estimate, no filter selectivity | Add B-tree index on filter column |
+| `Nested Loop` with large outer set | Exponential row growth in inner loop | Consider Hash Join; index inner join key |
+| `cost=... rows=1` but actual rows=50000 | Stale statistics | Run `ANALYZE <table>;` |
+| `Buffers: hit=10 read=90000` | Low buffer cache hit rate | Increase `shared_buffers`; add covering index |
+| `Sort Method: external merge` | Sort spilling to disk | Increase `work_mem` for the session |
+
+### Create a Covering Index
+```sql
+-- Covers the filter AND the projected columns, eliminating a heap fetch
+CREATE INDEX CONCURRENTLY idx_orders_status_created_covering
+    ON orders (status, created_at)
+    INCLUDE (customer_id, total_amount);
+```
+
+### Validate Improvement
+```sql
+-- Before optimization: save plan & timing
+EXPLAIN (ANALYZE, BUFFERS) <query>;   -- note "Execution Time: X ms"
+
+-- After optimization: compare
+EXPLAIN (ANALYZE, BUFFERS) <query>;   -- target meaningful reduction in cost & time
+
+-- Confirm index is actually used
+SELECT indexname, idx_scan, idx_tup_read, idx_tup_fetch
+FROM   pg_stat_user_indexes
+WHERE  relname = 'orders';
+```
+
+### MySQL: Find Slow Queries
+```sql
+-- Inspect slow query log candidates
+SELECT * FROM performance_schema.events_statements_summary_by_digest
+ORDER  BY SUM_TIMER_WAIT DESC
+LIMIT  20;
+
+-- Execution plan
+EXPLAIN FORMAT=JSON
+SELECT * FROM orders WHERE status = 'pending' AND created_at > NOW() - INTERVAL 7 DAY;
+```
+
+## Constraints
+
+### MUST DO
+- Capture `EXPLAIN (ANALYZE, BUFFERS)` output **before** optimizing — this is the baseline
+- Measure performance before and after every change
+- Create indexes with `CONCURRENTLY` (PostgreSQL) to avoid table locks
+- Test in non-production; roll back if write performance or replication lag worsens
+- Document all optimization decisions with before/after metrics
+- Run `ANALYZE` after bulk data changes to refresh statistics
+
+### MUST NOT DO
+- Apply optimizations without a measured baseline
+- Create redundant or unused indexes
+- Make multiple changes simultaneously (impossible to attribute impact)
+- Ignore write amplification caused by new indexes
+- Neglect `VACUUM` / statistics maintenance
+
+## Output Templates
+
+When optimizing database performance, provide:
+1. Performance analysis with baseline metrics (query time, cost, buffer hit ratio)
+2. Identified bottlenecks and root causes (with EXPLAIN evidence)
+3. Optimization strategy with specific changes
+4. Implementation SQL / config changes
+5. Validation queries to measure improvement
+6. Monitoring recommendations

package/skills/database/database-optimizer/references/index-strategies.md

@@ -0,0 +1,331 @@
+# Index Strategies
+
+## Index Selection Methodology
+
+### Identify Index Candidates
+
+```sql
+-- PostgreSQL: Find queries missing indexes
+SELECT query, calls, total_exec_time, mean_exec_time
+FROM pg_stat_statements
+WHERE mean_exec_time > 100
+ORDER BY total_exec_time DESC
+LIMIT 20;
+
+-- PostgreSQL: Find sequential scans on large tables
+SELECT schemaname, tablename, seq_scan, seq_tup_read,
+       idx_scan, seq_tup_read / seq_scan as avg_seq_tup_read
+FROM pg_stat_user_tables
+WHERE seq_scan > 0
+  AND seq_tup_read / seq_scan > 10000
+ORDER BY seq_tup_read DESC;
+
+-- MySQL: Check table scans
+SELECT * FROM sys.statements_with_full_table_scans
+WHERE db = 'your_database'
+ORDER BY exec_count DESC;
+```
+
+## B-Tree Indexes (Default)
+
+### Single Column Indexes
+
+```sql
+-- Create index for WHERE clauses
+CREATE INDEX idx_users_email ON users(email);
+
+-- Create index for JOIN conditions
+CREATE INDEX idx_orders_user_id ON orders(user_id);
+
+-- Create index for ORDER BY
+CREATE INDEX idx_products_price ON products(price);
+
+-- Unique constraint as index
+CREATE UNIQUE INDEX idx_users_username ON users(username);
+```
+
+### Multi-Column Indexes
+
+```sql
+-- Order matters: most selective column first
+CREATE INDEX idx_orders_status_created
+ON orders(status, created_at);
+
+-- Good for queries:
+-- WHERE status = 'pending'
+-- WHERE status = 'pending' AND created_at > '2024-01-01'
+-- WHERE status = 'pending' ORDER BY created_at
+
+-- NOT good for:
+-- WHERE created_at > '2024-01-01' (status not specified)
+
+-- Include commonly queried columns
+CREATE INDEX idx_users_active_email_name
+ON users(active, email) INCLUDE (name);
+```
+
+### Column Order Guidelines
+
+```sql
+-- Rule 1: Equality before range
+CREATE INDEX idx_events_type_timestamp
+ON events(type, timestamp);  -- type = 'click' AND timestamp > ...
+
+-- Rule 2: High selectivity first
+CREATE INDEX idx_orders_user_status
+ON orders(user_id, status);  -- user_id is more selective than status
+
+-- Rule 3: Match query patterns
+-- Query: WHERE country = 'US' AND city = 'NYC' AND zip = '10001'
+CREATE INDEX idx_locations_country_city_zip
+ON locations(country, city, zip);
+```
+
+## Covering Indexes
+
+### PostgreSQL INCLUDE Clause
+
+```sql
+-- Include non-key columns for index-only scans
+CREATE INDEX idx_users_email_covering
+ON users(email) INCLUDE (name, created_at);
+
+-- Query can be satisfied entirely from index
+EXPLAIN (ANALYZE, BUFFERS)
+SELECT name, created_at
+FROM users
+WHERE email = 'user@example.com';
+-- Should show "Index Only Scan"
+```
+
+### MySQL Covering Indexes
+
+```sql
+-- MySQL: Add columns to end of index
+CREATE INDEX idx_orders_user_covering
+ON orders(user_id, status, created_at, total);
+
+-- Query uses covering index
+EXPLAIN
+SELECT status, created_at, total
+FROM orders
+WHERE user_id = 123;
+-- Should show "Using index" in Extra column
+```
+
+## Partial Indexes
+
+### PostgreSQL Partial Indexes
+
+```sql
+-- Index only active users
+CREATE INDEX idx_users_active_email
+ON users(email)
+WHERE active = true;
+
+-- Index only recent orders
+CREATE INDEX idx_orders_recent
+ON orders(user_id, created_at)
+WHERE created_at > NOW() - INTERVAL '30 days';
+
+-- Index only pending/processing orders (ignore completed)
+CREATE INDEX idx_orders_active
+ON orders(status, user_id)
+WHERE status IN ('pending', 'processing');
+
+-- Smaller index = better performance + less storage
+```
+
+### MySQL Filtered Indexes (8.0+)
+
+```sql
+-- MySQL 8.0+ supports functional indexes for similar effect
+CREATE INDEX idx_users_active
+ON users((CASE WHEN active = 1 THEN email END));
+```
+
+## Expression Indexes
+
+### PostgreSQL Function Indexes
+
+```sql
+-- Index for case-insensitive search
+CREATE INDEX idx_users_email_lower
+ON users(LOWER(email));
+
+-- Query must match expression
+SELECT * FROM users
+WHERE LOWER(email) = LOWER('User@Example.com');
+
+-- Index for JSONB queries
+CREATE INDEX idx_users_settings_theme
+ON users((settings->>'theme'));
+
+SELECT * FROM users
+WHERE settings->>'theme' = 'dark';
+
+-- Index for date truncation
+CREATE INDEX idx_orders_date
+ON orders(DATE(created_at));
+```
+
+### MySQL Generated Column Indexes
+
+```sql
+-- Create generated column, then index it
+ALTER TABLE users
+ADD COLUMN email_lower VARCHAR(255)
+GENERATED ALWAYS AS (LOWER(email)) STORED;
+
+CREATE INDEX idx_users_email_lower
+ON users(email_lower);
+
+-- Use in queries
+SELECT * FROM users
+WHERE email_lower = LOWER('User@Example.com');
+```
+
+## Specialized Index Types
+
+### PostgreSQL GIN Indexes (Full-Text, Arrays, JSONB)
+
+```sql
+-- Full-text search
+CREATE INDEX idx_posts_search
+ON posts USING GIN(to_tsvector('english', title || ' ' || content));
+
+SELECT * FROM posts
+WHERE to_tsvector('english', title || ' ' || content)
+      @@ to_tsquery('english', 'database & optimization');
+
+-- Array search
+CREATE INDEX idx_products_tags
+ON products USING GIN(tags);
+
+SELECT * FROM products
+WHERE tags @> ARRAY['electronics', 'sale'];
+
+-- JSONB search
+CREATE INDEX idx_users_metadata
+ON users USING GIN(metadata);
+
+SELECT * FROM users
+WHERE metadata @> '{"plan": "premium"}';
+```
+
+### PostgreSQL GiST Indexes (Geometric, Range)
+
+```sql
+-- Range types
+CREATE INDEX idx_events_time_range
+ON events USING GIST(time_range);
+
+SELECT * FROM events
+WHERE time_range && '[2024-01-01, 2024-01-31]'::tstzrange;
+
+-- PostGIS geometric queries
+CREATE INDEX idx_locations_coords
+ON locations USING GIST(coordinates);
+```
+
+### MySQL Full-Text Indexes
+
+```sql
+-- Full-text search
+CREATE FULLTEXT INDEX idx_posts_content
+ON posts(title, content);
+
+SELECT * FROM posts
+WHERE MATCH(title, content)
+      AGAINST('database optimization' IN NATURAL LANGUAGE MODE);
+
+-- Boolean mode for complex searches
+SELECT * FROM posts
+WHERE MATCH(title, content)
+      AGAINST('+database -mysql' IN BOOLEAN MODE);
+```
+
+## Index Maintenance
+
+### PostgreSQL Maintenance
+
+```sql
+-- Update statistics for query planner
+ANALYZE users;
+
+-- Rebuild bloated index
+REINDEX INDEX CONCURRENTLY idx_users_email;
+
+-- Check index bloat
+SELECT
+    schemaname, tablename, indexname,
+    pg_size_pretty(pg_relation_size(indexrelid)) as index_size,
+    idx_scan as scans,
+    idx_tup_read as tuples_read,
+    idx_tup_fetch as tuples_fetched
+FROM pg_stat_user_indexes
+ORDER BY pg_relation_size(indexrelid) DESC;
+
+-- Find unused indexes
+SELECT
+    schemaname, tablename, indexname,
+    idx_scan,
+    pg_size_pretty(pg_relation_size(indexrelid)) as index_size
+FROM pg_stat_user_indexes
+WHERE idx_scan = 0
+  AND indexrelname NOT LIKE 'pg_toast%'
+ORDER BY pg_relation_size(indexrelid) DESC;
+```
+
+### MySQL Maintenance
+
+```sql
+-- Update statistics
+ANALYZE TABLE users;
+
+-- Rebuild index
+ALTER TABLE users DROP INDEX idx_users_email, ADD INDEX idx_users_email(email);
+
+-- Check index usage
+SELECT
+    object_schema,
+    object_name,
+    index_name,
+    count_star,
+    count_read,
+    count_fetch
+FROM performance_schema.table_io_waits_summary_by_index_usage
+WHERE object_schema = 'your_database'
+ORDER BY count_star DESC;
+
+-- Find unused indexes
+SELECT
+    object_schema,
+    object_name,
+    index_name
+FROM performance_schema.table_io_waits_summary_by_index_usage
+WHERE index_name IS NOT NULL
+  AND count_star = 0
+  AND object_schema = 'your_database';
+```
+
+## Index Anti-Patterns
+
+| Anti-Pattern | Issue | Solution |
+|-------------|-------|----------|
+| Index every column | Write overhead, storage waste | Index based on query patterns |
+| Redundant indexes | `(a)` + `(a,b)` | Keep only `(a,b)` |
+| Wrong column order | `(created_at, user_id)` for `WHERE user_id = ?` | Put filtered columns first |
+| Over-covering | Including rarely-used columns | Include only frequently accessed columns |
+| Ignoring WHERE clause | Full index for 5% of data | Use partial indexes |
+| Expression mismatch | Index `email`, query `LOWER(email)` | Create expression index |
+
+## Index Design Checklist
+
+1. **Analyze queries**: Use pg_stat_statements or slow query log
+2. **Check execution plans**: Look for Seq Scan on large tables
+3. **Design indexes**: Equality → Range → Include
+4. **Create concurrently**: Avoid locking (PostgreSQL)
+5. **Validate improvement**: Compare before/after EXPLAIN
+6. **Monitor usage**: Remove unused indexes after 30 days
+7. **Maintain regularly**: VACUUM, ANALYZE, REINDEX as needed