aigroup-workflow 2.1.2 → 2.2.0

package/skills/database/postgres-pro/SKILL.md

@@ -0,0 +1,152 @@
+---
+name: postgres-pro
+description: Use when optimizing PostgreSQL queries, configuring replication, or implementing advanced database features. Invoke for EXPLAIN analysis, JSONB operations, extension usage, VACUUM tuning, performance monitoring.
+license: MIT
+metadata:
+  author: https://github.com/Jeffallan
+  version: "1.1.0"
+  domain: infrastructure
+  triggers: PostgreSQL, Postgres, EXPLAIN ANALYZE, pg_stat, JSONB, streaming replication, logical replication, VACUUM, PostGIS, pgvector
+  role: specialist
+  scope: implementation
+  output-format: code
+  related-skills: database-optimizer, devops-engineer, sre-engineer
+---
+
+# PostgreSQL Pro
+
+Senior PostgreSQL expert with deep expertise in database administration, performance optimization, and advanced PostgreSQL features.
+
+## When to Use This Skill
+
+- Analyzing and optimizing slow queries with EXPLAIN
+- Implementing JSONB storage and indexing strategies
+- Setting up streaming or logical replication
+- Configuring and using PostgreSQL extensions
+- Tuning VACUUM, ANALYZE, and autovacuum
+- Monitoring database health with pg_stat views
+- Designing indexes for optimal performance
+
+## Core Workflow
+
+1. **Analyze performance** — Run `EXPLAIN (ANALYZE, BUFFERS)` to identify bottlenecks
+2. **Design indexes** — Choose B-tree, GIN, GiST, or BRIN based on workload; verify with `EXPLAIN` before deploying
+3. **Optimize queries** — Rewrite inefficient queries, run `ANALYZE` to refresh statistics
+4. **Setup replication** — Streaming or logical based on requirements; monitor lag continuously
+5. **Monitor and maintain** — Track VACUUM, bloat, and autovacuum via `pg_stat` views; verify improvements after each change
+
+### End-to-End Example: Slow Query → Fix → Verification
+
+```sql
+-- Step 1: Identify slow queries
+SELECT query, mean_exec_time, calls
+FROM pg_stat_statements
+ORDER BY mean_exec_time DESC
+LIMIT 10;
+
+-- Step 2: Analyze a specific slow query
+EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
+SELECT * FROM orders WHERE customer_id = 42 AND status = 'pending';
+-- Look for: Seq Scan (bad on large tables), high Buffers hit, nested loops on large sets
+
+-- Step 3: Create a targeted index
+CREATE INDEX CONCURRENTLY idx_orders_customer_status
+  ON orders (customer_id, status)
+  WHERE status = 'pending';  -- partial index reduces size
+
+-- Step 4: Verify the index is used
+EXPLAIN (ANALYZE, BUFFERS)
+SELECT * FROM orders WHERE customer_id = 42 AND status = 'pending';
+-- Confirm: Index Scan on idx_orders_customer_status, lower actual time
+
+-- Step 5: Update statistics if needed after bulk changes
+ANALYZE orders;
+```
+
+## Reference Guide
+
+Load detailed guidance based on context:
+
+| Topic | Reference | Load When |
+|-------|-----------|-----------|
+| Performance | `references/performance.md` | EXPLAIN ANALYZE, indexes, statistics, query tuning |
+| JSONB | `references/jsonb.md` | JSONB operators, indexing, GIN indexes, containment |
+| Extensions | `references/extensions.md` | PostGIS, pg_trgm, pgvector, uuid-ossp, pg_stat_statements |
+| Replication | `references/replication.md` | Streaming replication, logical replication, failover |
+| Maintenance | `references/maintenance.md` | VACUUM, ANALYZE, pg_stat views, monitoring, bloat |
+
+## Common Patterns
+
+### JSONB — GIN Index and Query
+
+```sql
+-- Create GIN index for containment queries
+CREATE INDEX idx_events_payload ON events USING GIN (payload);
+
+-- Efficient JSONB containment query (uses GIN index)
+SELECT * FROM events WHERE payload @> '{"type": "login", "success": true}';
+
+-- Extract nested value
+SELECT payload->>'user_id', payload->'meta'->>'ip'
+FROM events
+WHERE payload @> '{"type": "login"}';
+```
+
+### VACUUM and Bloat Monitoring
+
+```sql
+-- Check tables with high dead tuple counts
+SELECT relname, n_dead_tup, n_live_tup,
+       round(n_dead_tup::numeric / NULLIF(n_live_tup + n_dead_tup, 0) * 100, 2) AS dead_pct,
+       last_autovacuum
+FROM pg_stat_user_tables
+ORDER BY n_dead_tup DESC
+LIMIT 20;
+
+-- Manually vacuum a high-churn table and verify
+VACUUM (ANALYZE, VERBOSE) orders;
+```
+
+### Replication Lag Monitoring
+
+```sql
+-- On primary: check standby lag
+SELECT client_addr, state, sent_lsn, write_lsn, flush_lsn, replay_lsn,
+       (sent_lsn - replay_lsn) AS replication_lag_bytes
+FROM pg_stat_replication;
+```
+
+## Constraints
+
+### MUST DO
+- Use `EXPLAIN (ANALYZE, BUFFERS)` for query optimization
+- Verify indexes are actually used with `EXPLAIN` before and after creation
+- Use `CREATE INDEX CONCURRENTLY` to avoid table locks in production
+- Run `ANALYZE` after bulk data changes to refresh statistics
+- Monitor autovacuum; tune `autovacuum_vacuum_scale_factor` for high-churn tables
+- Use connection pooling (pgBouncer, pgPool)
+- Monitor replication lag via `pg_stat_replication`
+- Use prepared statements to prevent SQL injection
+- Use `uuid` type for UUIDs, not `text`
+
+### MUST NOT DO
+- Disable autovacuum globally
+- Create indexes without first analyzing query patterns
+- Use `SELECT *` in production queries
+- Ignore replication lag alerts
+- Skip VACUUM on high-churn tables
+- Store large BLOBs in the database (use object storage)
+- Deploy index changes without verifying the planner uses them
+
+## Output Templates
+
+When implementing PostgreSQL solutions, provide:
+1. Query with `EXPLAIN (ANALYZE, BUFFERS)` output and interpretation
+2. Index definitions with rationale and pre/post verification
+3. Configuration changes with before/after values
+4. Monitoring queries for ongoing health checks
+5. Brief explanation of performance impact
+
+## Knowledge Reference
+
+PostgreSQL 12-16, EXPLAIN ANALYZE, B-tree/GIN/GiST/BRIN indexes, JSONB operators, streaming replication, logical replication, VACUUM/ANALYZE, pg_stat views, PostGIS, pgvector, pg_trgm, WAL archiving, PITR

package/skills/database/postgres-pro/references/extensions.md

@@ -0,0 +1,404 @@
+# PostgreSQL Extensions
+
+## Extension Management
+
+```sql
+-- List available extensions
+SELECT * FROM pg_available_extensions ORDER BY name;
+
+-- List installed extensions
+SELECT * FROM pg_extension;
+
+-- Install extension
+CREATE EXTENSION IF NOT EXISTS pg_stat_statements;
+CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
+CREATE EXTENSION IF NOT EXISTS pg_trgm;
+
+-- Drop extension
+DROP EXTENSION pg_stat_statements;
+
+-- Update extension
+ALTER EXTENSION pg_stat_statements UPDATE TO '1.10';
+```
+
+## pg_stat_statements (Query Performance)
+
+```sql
+-- Install and configure
+CREATE EXTENSION pg_stat_statements;
+
+-- postgresql.conf:
+-- shared_preload_libraries = 'pg_stat_statements'
+-- pg_stat_statements.max = 10000
+-- pg_stat_statements.track = all
+
+-- Top 10 slowest queries by mean time
+SELECT
+  query,
+  calls,
+  total_exec_time,
+  mean_exec_time,
+  max_exec_time,
+  stddev_exec_time,
+  rows
+FROM pg_stat_statements
+ORDER BY mean_exec_time DESC
+LIMIT 10;
+
+-- Most frequently called queries
+SELECT
+  query,
+  calls,
+  total_exec_time,
+  mean_exec_time
+FROM pg_stat_statements
+ORDER BY calls DESC
+LIMIT 10;
+
+-- Most time-consuming queries (total time)
+SELECT
+  query,
+  calls,
+  total_exec_time / 1000 as total_seconds,
+  mean_exec_time,
+  (total_exec_time / sum(total_exec_time) OVER ()) * 100 as percentage
+FROM pg_stat_statements
+ORDER BY total_exec_time DESC
+LIMIT 10;
+
+-- Reset statistics
+SELECT pg_stat_statements_reset();
+```
+
+## uuid-ossp (UUID Generation)
+
+```sql
+CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
+
+-- Generate UUIDs
+SELECT uuid_generate_v1();    -- Time-based + MAC address
+SELECT uuid_generate_v4();    -- Random (most common)
+
+-- Use in tables
+CREATE TABLE users (
+  id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
+  email TEXT NOT NULL,
+  created_at TIMESTAMPTZ DEFAULT NOW()
+);
+
+-- Insert
+INSERT INTO users (email) VALUES ('user@example.com')
+RETURNING id;
+```
+
+## pg_trgm (Fuzzy String Matching)
+
+```sql
+CREATE EXTENSION IF NOT EXISTS pg_trgm;
+
+-- Similarity search
+SELECT
+  email,
+  similarity(email, 'john@example.com') as sim
+FROM users
+WHERE similarity(email, 'john@example.com') > 0.3
+ORDER BY sim DESC;
+
+-- LIKE optimization with trigram index
+CREATE INDEX idx_users_email_trgm ON users USING GIN(email gin_trgm_ops);
+
+-- Now these queries use index:
+SELECT * FROM users WHERE email ILIKE '%john%';
+SELECT * FROM users WHERE email % 'jon@example.com';  -- Similar to
+
+-- Trigram operators
+SELECT 'hello' % 'helo';              -- True (similar)
+SELECT similarity('hello', 'helo');   -- 0.5
+SELECT word_similarity('hello', 'hello world');  -- 1.0
+
+-- Set similarity threshold
+SET pg_trgm.similarity_threshold = 0.5;
+SELECT * FROM users WHERE email % 'searchtext';
+```
+
+## PostGIS (Spatial and Geographic)
+
+```sql
+CREATE EXTENSION IF NOT EXISTS postgis;
+
+-- Create spatial table
+CREATE TABLE locations (
+  id SERIAL PRIMARY KEY,
+  name TEXT NOT NULL,
+  geom GEOMETRY(Point, 4326)  -- WGS84 lat/lng
+);
+
+-- Add spatial index
+CREATE INDEX idx_locations_geom ON locations USING GIST(geom);
+
+-- Insert point (longitude, latitude)
+INSERT INTO locations (name, geom)
+VALUES ('NYC', ST_SetSRID(ST_MakePoint(-74.0060, 40.7128), 4326));
+
+-- Distance queries (in meters)
+SELECT
+  name,
+  ST_Distance(
+    geom::geography,
+    ST_SetSRID(ST_MakePoint(-73.9857, 40.7484), 4326)::geography
+  ) as distance_meters
+FROM locations
+ORDER BY distance_meters
+LIMIT 10;
+
+-- Within radius (1km = 1000m)
+SELECT * FROM locations
+WHERE ST_DWithin(
+  geom::geography,
+  ST_SetSRID(ST_MakePoint(-74.0060, 40.7128), 4326)::geography,
+  1000
+);
+
+-- Bounding box query (very fast with GIST index)
+SELECT * FROM locations
+WHERE geom && ST_MakeEnvelope(-74.1, 40.6, -73.9, 40.8, 4326);
+
+-- Contains query
+SELECT * FROM zones
+WHERE ST_Contains(geom, ST_SetSRID(ST_MakePoint(-74.0060, 40.7128), 4326));
+
+-- Area calculation
+SELECT
+  name,
+  ST_Area(geom::geography) / 1000000 as area_km2
+FROM zones;
+
+-- GeoJSON export
+SELECT
+  name,
+  ST_AsGeoJSON(geom) as geojson
+FROM locations;
+```
+
+## pgvector (Vector Similarity Search)
+
+```sql
+CREATE EXTENSION IF NOT EXISTS vector;
+
+-- Create table with vector column
+CREATE TABLE embeddings (
+  id SERIAL PRIMARY KEY,
+  content TEXT,
+  embedding vector(1536)  -- OpenAI embeddings are 1536 dimensions
+);
+
+-- Add vector index (HNSW for better performance)
+CREATE INDEX ON embeddings USING hnsw (embedding vector_cosine_ops);
+-- or IVFFlat for memory efficiency:
+-- CREATE INDEX ON embeddings USING ivfflat (embedding vector_cosine_ops);
+
+-- Insert vectors
+INSERT INTO embeddings (content, embedding)
+VALUES ('Hello world', '[0.1, 0.2, 0.3, ...]');
+
+-- Similarity search (cosine distance)
+SELECT
+  content,
+  1 - (embedding <=> '[0.1, 0.2, ...]') as similarity
+FROM embeddings
+ORDER BY embedding <=> '[0.1, 0.2, ...]'
+LIMIT 10;
+
+-- Distance operators
+-- <-> L2 distance (Euclidean)
+-- <#> negative inner product
+-- <=> cosine distance (most common for embeddings)
+
+-- Set index parameters for better recall
+SET hnsw.ef_search = 100;  -- Higher = better recall, slower query
+
+-- Bulk insert optimization
+SET maintenance_work_mem = '2GB';
+```
+
+## pgcrypto (Encryption and Hashing)
+
+```sql
+CREATE EXTENSION IF NOT EXISTS pgcrypto;
+
+-- Hash passwords (use bcrypt)
+INSERT INTO users (email, password_hash)
+VALUES ('user@example.com', crypt('password123', gen_salt('bf', 10)));
+
+-- Verify password
+SELECT * FROM users
+WHERE email = 'user@example.com'
+  AND password_hash = crypt('password123', password_hash);
+
+-- Generate random values
+SELECT gen_random_uuid();
+SELECT gen_random_bytes(32);
+
+-- Encrypt/decrypt data
+SELECT
+  pgp_sym_encrypt('sensitive data', 'encryption-key'),
+  pgp_sym_decrypt(encrypted_column, 'encryption-key')
+FROM table_name;
+
+-- Digest functions
+SELECT digest('data', 'sha256');
+SELECT encode(digest('data', 'sha256'), 'hex');
+```
+
+## postgres_fdw (Foreign Data Wrapper)
+
+```sql
+CREATE EXTENSION IF NOT EXISTS postgres_fdw;
+
+-- Create foreign server
+CREATE SERVER remote_db
+FOREIGN DATA WRAPPER postgres_fdw
+OPTIONS (host 'remote-host', port '5432', dbname 'remote_db');
+
+-- User mapping
+CREATE USER MAPPING FOR current_user
+SERVER remote_db
+OPTIONS (user 'remote_user', password 'remote_password');
+
+-- Import foreign schema
+IMPORT FOREIGN SCHEMA public
+FROM SERVER remote_db
+INTO remote_schema;
+
+-- Or create specific foreign table
+CREATE FOREIGN TABLE remote_users (
+  id INTEGER,
+  email TEXT,
+  created_at TIMESTAMPTZ
+)
+SERVER remote_db
+OPTIONS (schema_name 'public', table_name 'users');
+
+-- Query remote table (transparent)
+SELECT * FROM remote_users WHERE created_at > NOW() - INTERVAL '1 day';
+
+-- Join local and remote tables
+SELECT
+  l.id,
+  l.name,
+  r.email
+FROM local_table l
+JOIN remote_users r ON l.user_id = r.id;
+```
+
+## pg_repack (Online Table Reorganization)
+
+```sql
+CREATE EXTENSION IF NOT EXISTS pg_repack;
+
+-- Repack table (removes bloat, rebuilds indexes)
+-- Run via command line, not SQL:
+-- pg_repack -d mydb -t users
+
+-- Check bloat before repack
+SELECT
+  schemaname,
+  tablename,
+  pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) as size,
+  pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename) -
+                 pg_relation_size(schemaname||'.'||tablename)) as index_size
+FROM pg_tables
+WHERE schemaname = 'public'
+ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC;
+
+-- Repack entire database
+-- pg_repack -d mydb
+
+-- Repack with custom order
+-- pg_repack -d mydb -t users -o "created_at DESC"
+```
+
+## timescaledb (Time-Series Data)
+
+```sql
+CREATE EXTENSION IF NOT EXISTS timescaledb;
+
+-- Create hypertable (must have time column)
+CREATE TABLE metrics (
+  time TIMESTAMPTZ NOT NULL,
+  device_id INTEGER,
+  temperature DOUBLE PRECISION,
+  humidity DOUBLE PRECISION
+);
+
+-- Convert to hypertable
+SELECT create_hypertable('metrics', 'time');
+
+-- Set chunk interval (default 7 days)
+SELECT set_chunk_time_interval('metrics', INTERVAL '1 day');
+
+-- Add compression
+ALTER TABLE metrics SET (
+  timescaledb.compress,
+  timescaledb.compress_segmentby = 'device_id',
+  timescaledb.compress_orderby = 'time DESC'
+);
+
+-- Automatic compression policy (compress chunks older than 7 days)
+SELECT add_compression_policy('metrics', INTERVAL '7 days');
+
+-- Retention policy (drop chunks older than 30 days)
+SELECT add_retention_policy('metrics', INTERVAL '30 days');
+
+-- Continuous aggregates (materialized views for time-series)
+CREATE MATERIALIZED VIEW metrics_hourly
+WITH (timescaledb.continuous) AS
+SELECT
+  time_bucket('1 hour', time) AS bucket,
+  device_id,
+  AVG(temperature) as avg_temp,
+  MAX(temperature) as max_temp,
+  MIN(temperature) as min_temp
+FROM metrics
+GROUP BY bucket, device_id;
+
+-- Refresh policy
+SELECT add_continuous_aggregate_policy('metrics_hourly',
+  start_offset => INTERVAL '3 hours',
+  end_offset => INTERVAL '1 hour',
+  schedule_interval => INTERVAL '1 hour'
+);
+```
+
+## Extension Recommendations by Use Case
+
+**Query Performance Monitoring:**
+- `pg_stat_statements` (essential)
+- `pg_stat_kcache` (cache hit statistics)
+
+**Text Search:**
+- `pg_trgm` (fuzzy matching, LIKE optimization)
+- Built-in full-text search (no extension needed)
+
+**Spatial Data:**
+- `postgis` (comprehensive spatial features)
+
+**Vector Embeddings / AI:**
+- `pgvector` (for semantic search, RAG applications)
+
+**Time-Series:**
+- `timescaledb` (automatic partitioning, compression)
+
+**Data Security:**
+- `pgcrypto` (hashing, encryption)
+- `pg_audit` (audit logging)
+
+**UUID Support:**
+- `uuid-ossp` (UUID generation)
+
+**Cross-Database Queries:**
+- `postgres_fdw` (query remote PostgreSQL)
+- `file_fdw` (query CSV files)
+
+**Table Maintenance:**
+- `pg_repack` (online bloat removal)