@jetrabbits/agentic 0.0.1

package/areas/devops/database-ops/skills/backup-restore/SKILL.md

@@ -0,0 +1,226 @@
+---
+name: backup-restore
+type: skill
+description: PostgreSQL backup and restore with pgBackRest — full/incremental/WAL, PITR, K8s CronJob scheduling, and restore verification.
+related-rules:
+  - backup-policy.md
+allowed-tools: Read, Write, Edit, Bash
+---
+
+# Skill: Backup & Restore
+
+> **Expertise:** pgBackRest full/WAL/incremental backup, PITR, S3/MinIO storage, CronJob scheduling, CloudNativePG backup CRDs.
+
+## When to load
+
+When configuring backup infrastructure, running a restore, verifying backup integrity, or recovering from data loss.
+
+## pgBackRest Configuration
+
+```ini
+# /etc/pgbackrest/pgbackrest.conf
+[global]
+repo1-type=s3
+repo1-path=/postgres-backups
+repo1-s3-bucket=mycompany-db-backups
+repo1-s3-endpoint=s3.eu-west-1.amazonaws.com
+repo1-s3-region=eu-west-1
+repo1-s3-key=<AWS_ACCESS_KEY>
+repo1-s3-key-secret=<AWS_SECRET_KEY>
+repo1-cipher-type=aes-256-cbc
+repo1-cipher-pass=<STRONG_PASSPHRASE>    # rotate annually
+repo1-retention-full=4                  # keep 4 full backups
+repo1-retention-diff=14                 # keep 14 differential backups
+
+[global:archive-push]
+compress-level=3
+
+[production-db]
+pg1-path=/var/lib/postgresql/data
+pg1-host=postgres-primary
+pg1-host-user=postgres
+```
+
+## PostgreSQL: Enable WAL Archiving
+
+```sql
+-- postgresql.conf (or via ALTER SYSTEM)
+ALTER SYSTEM SET wal_level = replica;
+ALTER SYSTEM SET archive_mode = on;
+ALTER SYSTEM SET archive_command = 'pgbackrest --stanza=production-db archive-push %p';
+ALTER SYSTEM SET archive_timeout = '300';   -- archive segment at least every 5 min
+SELECT pg_reload_conf();
+```
+
+## Backup Commands
+
+```bash
+# Initialize stanza (run once)
+pgbackrest --stanza=production-db stanza-create
+
+# Full backup (schedule: weekly)
+pgbackrest --stanza=production-db --type=full backup
+
+# Differential backup (changes since last full — schedule: daily)
+pgbackrest --stanza=production-db --type=diff backup
+
+# Incremental backup (changes since last backup — schedule: hourly)
+pgbackrest --stanza=production-db --type=incr backup
+
+# Check backup catalog
+pgbackrest --stanza=production-db info
+
+# Verify backup integrity (reads and checksums)
+pgbackrest --stanza=production-db --set=<backup-label> check
+```
+
+## PITR Restore (Point-in-Time Recovery)
+
+```bash
+# PITR to specific timestamp (e.g., 1 minute before accidental DELETE)
+# Step 1: Stop PostgreSQL
+systemctl stop postgresql
+
+# Step 2: Restore
+pgbackrest --stanza=production-db \
+  --delta \
+  --target="2024-11-15 03:40:00+00" \
+  --target-action=promote \
+  --target-timeline=current \
+  restore
+
+# Step 3: Start PostgreSQL in recovery mode
+systemctl start postgresql
+
+# Step 4: Monitor recovery progress
+tail -f /var/log/postgresql/postgresql.log
+# Watch for: "recovery stopping before commit of transaction"
+# Then:      "database system is ready to accept connections"
+
+# Step 5: Verify data integrity
+psql -c "SELECT count(*) FROM orders WHERE created_at > '2024-11-15 03:39:00';"
+```
+
+## K8s CronJob: Automated Backups
+
+```yaml
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: postgres-backup-daily
+  namespace: database
+spec:
+  schedule: "0 2 * * *"      # 02:00 UTC daily
+  concurrencyPolicy: Forbid
+  successfulJobsHistoryLimit: 3
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    spec:
+      template:
+        spec:
+          serviceAccountName: postgres-backup
+          restartPolicy: OnFailure
+          containers:
+            - name: pgbackrest
+              image: pgbackrest/pgbackrest:2.50
+              command:
+                - /bin/sh
+                - -c
+                - |
+                  pgbackrest --stanza=production-db --type=diff backup
+                  pgbackrest --stanza=production-db check
+              envFrom:
+                - secretRef:
+                    name: pgbackrest-s3-credentials
+              volumeMounts:
+                - name: pgbackrest-config
+                  mountPath: /etc/pgbackrest
+          volumes:
+            - name: pgbackrest-config
+              configMap:
+                name: pgbackrest-config
+```
+
+## CloudNativePG Backup (K8s Operator)
+
+```yaml
+# Cluster with scheduled backup
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: postgres-cluster
+spec:
+  backup:
+    barmanObjectStore:
+      destinationPath: s3://mycompany-db-backups/production
+      s3Credentials:
+        accessKeyId:
+          name: s3-creds
+          key: ACCESS_KEY_ID
+        secretAccessKey:
+          name: s3-creds
+          key: SECRET_ACCESS_KEY
+      wal:
+        compression: gzip
+      data:
+        compression: gzip
+    retentionPolicy: "30d"
+
+---
+# Scheduled backup CRD
+apiVersion: postgresql.cnpg.io/v1
+kind: ScheduledBackup
+metadata:
+  name: postgres-daily
+spec:
+  schedule: "0 2 * * *"
+  backupOwnerReference: self
+  cluster:
+    name: postgres-cluster
+```
+
+## Restore Verification Script (weekly)
+
+```bash
+#!/bin/bash
+# verify-backup.sh — restore to isolated pod and check row counts
+
+STANZA="production-db"
+RESTORE_HOST="postgres-restore-test"
+TABLES=("orders" "payments" "users")
+
+echo "=== pgBackRest Backup Verification $(date) ==="
+
+# 1. Check last backup age
+LAST_BACKUP=$(pgbackrest --stanza=$STANZA info --output=json | \
+  jq -r '.[] | .backup[-1].timestamp.stop')
+echo "Last backup: $LAST_BACKUP"
+
+AGE_HOURS=$(( ($(date +%s) - $(date -d "$LAST_BACKUP" +%s)) / 3600 ))
+if [ $AGE_HOURS -gt 26 ]; then
+  echo "ERROR: Last backup is ${AGE_HOURS}h old — exceeds 26h threshold"
+  exit 1
+fi
+
+# 2. Restore to test instance
+pgbackrest --stanza=$STANZA --delta --pg1-host=$RESTORE_HOST restore
+
+# 3. Start test PostgreSQL
+pg_ctl -D /var/lib/postgresql/restore -l /tmp/restore.log start
+
+# 4. Verify row counts match production
+for table in "${TABLES[@]}"; do
+  PROD_COUNT=$(psql -h postgres-primary -c "SELECT count(*) FROM $table;" -t | tr -d ' ')
+  REST_COUNT=$(psql -h $RESTORE_HOST -c "SELECT count(*) FROM $table;" -t | tr -d ' ')
+  if [ "$PROD_COUNT" != "$REST_COUNT" ]; then
+    echo "MISMATCH in $table: prod=$PROD_COUNT restore=$REST_COUNT"
+    FAILED=1
+  else
+    echo "OK: $table = $PROD_COUNT rows"
+  fi
+done
+
+pg_ctl -D /var/lib/postgresql/restore stop
+
+[ -z "$FAILED" ] && echo "BACKUP VERIFICATION PASSED" || exit 1
+```

package/areas/devops/database-ops/skills/db-performance/SKILL.md

@@ -0,0 +1,205 @@
+---
+name: db-performance
+type: skill
+description: PostgreSQL query performance — EXPLAIN ANALYZE, index design, pg_stat_statements, slow query detection, connection pool tuning.
+related-rules:
+  - access-control.md
+  - migration-runbook.md
+allowed-tools: Read, Bash
+---
+
+# Skill: Database Performance
+
+> **Expertise:** EXPLAIN ANALYZE, index design (partial/covering), pg_stat_statements, autovacuum tuning, PgBouncer sizing.
+
+## When to load
+
+When investigating slow queries, designing indexes, tuning PostgreSQL config, or sizing PgBouncer pools.
+
+## Query Analysis with pg_stat_statements
+
+```sql
+-- Enable (add to postgresql.conf, then restart or reload)
+-- shared_preload_libraries = 'pg_stat_statements'
+CREATE EXTENSION IF NOT EXISTS pg_stat_statements;
+
+-- Top 10 slowest queries by total time
+SELECT
+  left(query, 120) AS query_snippet,
+  calls,
+  round(total_exec_time::numeric, 2) AS total_ms,
+  round(mean_exec_time::numeric, 2)  AS mean_ms,
+  round(stddev_exec_time::numeric, 2) AS stddev_ms,
+  rows
+FROM pg_stat_statements
+ORDER BY total_exec_time DESC
+LIMIT 10;
+
+-- Top queries by mean execution time (find worst-per-call)
+SELECT
+  left(query, 120),
+  calls,
+  round(mean_exec_time::numeric, 2) AS mean_ms,
+  round(rows::numeric / calls, 1)   AS rows_per_call
+FROM pg_stat_statements
+WHERE calls > 100   -- ignore one-offs
+ORDER BY mean_exec_time DESC
+LIMIT 10;
+
+-- Queries with high I/O (missing index candidates)
+SELECT
+  left(query, 120),
+  calls,
+  round(mean_exec_time::numeric, 1) AS mean_ms,
+  shared_blks_read + shared_blks_hit AS total_blocks,
+  round(shared_blks_hit::numeric / NULLIF(shared_blks_hit + shared_blks_read, 0) * 100, 1) AS cache_hit_pct
+FROM pg_stat_statements
+WHERE calls > 50
+ORDER BY shared_blks_read DESC
+LIMIT 10;
+
+-- Reset stats after tuning
+SELECT pg_stat_statements_reset();
+```
+
+## EXPLAIN ANALYZE (reading execution plans)
+
+```sql
+-- Always use ANALYZE BUFFERS for real cost data
+EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
+SELECT o.*, c.email
+FROM orders o
+JOIN customers c ON c.id = o.customer_id
+WHERE o.status = 'pending'
+  AND o.created_at > now() - interval '7 days';
+
+/* Reading the output:
+   Seq Scan          → full table scan, may need index
+   Index Scan        → good, using index
+   Index Only Scan   → best, covering index (no heap access)
+   Nested Loop       → OK for small datasets; bad for large
+   Hash Join         → good for large joins
+   Merge Join        → good for pre-sorted data
+
+   Key numbers:
+   - actual time=START..END ms per row
+   - rows=N vs rows=N (estimated vs actual — big diff = stale stats)
+   - Buffers: hit=N read=N  (high 'read' = cache miss → index opportunity)
+*/
+```
+
+## Index Design Patterns
+
+```sql
+-- Standard B-tree (equality and range queries)
+CREATE INDEX CONCURRENTLY idx_orders_status_created
+  ON orders(status, created_at)
+  WHERE status IN ('pending', 'processing');   -- partial index — smaller, faster
+
+-- Covering index (index-only scan — no heap access)
+CREATE INDEX CONCURRENTLY idx_orders_customer_covering
+  ON orders(customer_id, created_at)
+  INCLUDE (status, total_amount);   -- INCLUDE avoids heap fetch for these columns
+
+-- Expression index (for function-based queries)
+CREATE INDEX CONCURRENTLY idx_users_email_lower
+  ON users(lower(email));           -- for: WHERE lower(email) = lower($1)
+
+-- JSON/JSONB index
+CREATE INDEX CONCURRENTLY idx_events_data_type
+  ON events USING GIN (data jsonb_path_ops);  -- for: WHERE data @> '{"type":"click"}'
+
+-- Multicolumn order: selectivity matters
+-- For: WHERE status='active' AND created_at > X
+-- ✅ (status, created_at) — filter on status first (low cardinality OK as prefix)
+-- ❌ (created_at, status) — date range first is wide; wastes I/O
+```
+
+## Identifying Missing Indexes
+
+```sql
+-- Tables with high sequential scans (candidates for indexing)
+SELECT
+  schemaname || '.' || relname AS table,
+  seq_scan,
+  seq_tup_read,
+  idx_scan,
+  round(seq_scan::numeric / NULLIF(seq_scan + idx_scan, 0) * 100, 1) AS seq_pct
+FROM pg_stat_user_tables
+WHERE seq_scan > 1000
+  AND seq_tup_read > 10000
+ORDER BY seq_tup_read DESC
+LIMIT 20;
+
+-- Unused indexes (wasting write overhead)
+SELECT
+  schemaname || '.' || tablename AS table,
+  indexname,
+  pg_size_pretty(pg_relation_size(indexrelid)) AS size,
+  idx_scan AS scans
+FROM pg_stat_user_indexes
+JOIN pg_index USING (indexrelid)
+WHERE idx_scan = 0
+  AND NOT indisunique          -- don't drop unique constraints
+  AND indpred IS NULL          -- don't drop partial indexes without analysis
+ORDER BY pg_relation_size(indexrelid) DESC;
+```
+
+## PostgreSQL Configuration Tuning
+
+```sql
+-- Key parameters for a 16GB RAM server
+ALTER SYSTEM SET shared_buffers = '4GB';          -- 25% of RAM
+ALTER SYSTEM SET effective_cache_size = '12GB';   -- 75% of RAM
+ALTER SYSTEM SET work_mem = '64MB';               -- per sort/hash; set conservatively
+ALTER SYSTEM SET maintenance_work_mem = '1GB';    -- for VACUUM, CREATE INDEX
+ALTER SYSTEM SET max_worker_processes = 8;
+ALTER SYSTEM SET max_parallel_workers = 4;
+ALTER SYSTEM SET max_parallel_workers_per_gather = 2;
+
+-- WAL tuning (for high-write workloads)
+ALTER SYSTEM SET wal_buffers = '64MB';
+ALTER SYSTEM SET checkpoint_completion_target = 0.9;
+ALTER SYSTEM SET min_wal_size = '1GB';
+ALTER SYSTEM SET max_wal_size = '4GB';
+
+SELECT pg_reload_conf();
+```
+
+## PgBouncer Pool Sizing
+
+```
+Formula:
+  max_server_connections = min(max_connections_pg - 5, available_connections)
+  default_pool_size = max_server_connections / number_of_services
+  max_client_conn = default_pool_size × 10   (clients can queue)
+
+Example (max_connections=200, 5 services):
+  default_pool_size = (200 - 5) / 5 = 39 → set to 40
+  max_client_conn = 400
+  reserve_pool_size = 5 (emergency burst)
+```
+
+```ini
+# pgbouncer.ini
+[pgbouncer]
+pool_mode = transaction          # best for stateless apps
+max_client_conn = 500            # total client connections
+default_pool_size = 40           # server connections per database+user
+reserve_pool_size = 5
+reserve_pool_timeout = 3
+server_idle_timeout = 600
+client_idle_timeout = 0
+query_wait_timeout = 30          # fail fast if no server available
+```
+
+## Autovacuum Tuning for High-Write Tables
+
+```sql
+-- Per-table autovacuum settings for hot tables
+ALTER TABLE orders SET (
+  autovacuum_vacuum_scale_factor = 0.01,     -- vacuum at 1% dead rows (default 20%)
+  autovacuum_analyze_scale_factor = 0.005,   -- analyze at 0.5% (default 10%)
+  autovacuum_vacuum_cost_delay = 2           -- less aggressive throttling
+);
+```

package/areas/devops/database-ops/skills/migration-safety/SKILL.md

@@ -0,0 +1,155 @@
+---
+name: migration-safety
+type: skill
+description: Safe database migrations in production — expand-and-contract, lock-safe DDL, timing estimation, rollback SQL.
+related-rules:
+  - migration-runbook.md
+  - backup-policy.md
+allowed-tools: Read, Write, Edit, Bash
+---
+
+# Skill: Migration Safety
+
+> **Expertise:** Expand-and-contract, `CREATE INDEX CONCURRENTLY`, migration timing estimation, rollback planning.
+
+## When to load
+
+When planning or executing a production database migration, estimating migration duration, or writing rollback SQL.
+
+## Expand-and-Contract Pattern
+
+```sql
+-- ❌ DANGEROUS: direct rename locks table and breaks old app version
+ALTER TABLE orders RENAME COLUMN user_id TO customer_id;
+
+-- ✅ SAFE: expand-and-contract over multiple deploys
+
+-- Phase 1: EXPAND (add new column, keep old)
+ALTER TABLE orders ADD COLUMN customer_id BIGINT;
+
+-- Phase 2: DUAL-WRITE (app v2 writes to both; reads from customer_id)
+-- (code change, no migration needed)
+
+-- Phase 3: BACKFILL (run in small batches to avoid lock)
+UPDATE orders SET customer_id = user_id
+WHERE customer_id IS NULL
+  AND id BETWEEN <batch_start> AND <batch_end>;
+
+-- Phase 4: CONTRACT (app v3 no longer uses user_id)
+ALTER TABLE orders DROP COLUMN user_id;
+```
+
+## Lock-Safe DDL
+
+```sql
+-- ✅ Safe: CREATE INDEX CONCURRENTLY (no table lock)
+CREATE INDEX CONCURRENTLY idx_orders_customer_id ON orders(customer_id);
+-- If concurrent creation fails:
+DROP INDEX CONCURRENTLY idx_orders_customer_id_invalid;
+-- Then retry
+
+-- ❌ Dangerous on large tables: full table lock
+CREATE INDEX idx_orders_customer_id ON orders(customer_id);
+
+-- ✅ Safe: ADD COLUMN with no default (instant in PostgreSQL 11+)
+ALTER TABLE orders ADD COLUMN processed_at TIMESTAMPTZ;
+
+-- ❌ Dangerous: ADD COLUMN with DEFAULT rewrites all rows (pre-PG11) / blocks (PG11+)
+ALTER TABLE orders ADD COLUMN processed_at TIMESTAMPTZ NOT NULL DEFAULT now();
+-- ✅ Safe alternative: add nullable, backfill, add constraint
+ALTER TABLE orders ADD COLUMN processed_at TIMESTAMPTZ;
+UPDATE orders SET processed_at = created_at WHERE processed_at IS NULL;
+ALTER TABLE orders ALTER COLUMN processed_at SET NOT NULL;
+```
+
+## Estimating Migration Duration
+
+```sql
+-- Estimate rows to process
+SELECT reltuples::BIGINT AS estimated_rows
+FROM pg_class
+WHERE relname = 'orders';
+
+-- Rough timing: ~100k rows/sec for simple UPDATE on indexed column
+-- ~10k rows/sec for complex JOIN-based UPDATE
+-- Always test on production-size staging first!
+
+-- Watch migration progress (PostgreSQL 9.6+)
+SELECT
+  phase,
+  blocks_done,
+  blocks_total,
+  round(100.0 * blocks_done / NULLIF(blocks_total, 0), 1) AS pct_done
+FROM pg_stat_progress_create_index
+WHERE relid = 'orders'::regclass;
+
+SELECT
+  phase,
+  tuples_done,
+  tuples_total,
+  round(100.0 * tuples_done / NULLIF(tuples_total, 0), 1) AS pct_done
+FROM pg_stat_progress_vacuum
+WHERE relid = 'orders'::regclass;
+```
+
+## Batched Backfill (avoid long transactions)
+
+```python
+# Never: UPDATE orders SET customer_id = user_id  (locks all rows, huge transaction)
+# Always: batch by primary key range
+
+import psycopg2
+
+BATCH_SIZE = 10_000
+
+with psycopg2.connect(DSN) as conn:
+    with conn.cursor() as cur:
+        cur.execute("SELECT MIN(id), MAX(id) FROM orders WHERE customer_id IS NULL")
+        min_id, max_id = cur.fetchone()
+
+    batch_start = min_id
+    while batch_start <= max_id:
+        batch_end = batch_start + BATCH_SIZE
+        with conn.cursor() as cur:
+            cur.execute("""
+                UPDATE orders
+                SET customer_id = user_id
+                WHERE id >= %s AND id < %s AND customer_id IS NULL
+            """, (batch_start, batch_end))
+        conn.commit()  # commit each batch — releases row locks
+        print(f"Backfilled {batch_start}–{batch_end}")
+        batch_start = batch_end
+```
+
+## Rollback SQL Template
+
+```sql
+-- Every migration should have a paired rollback script
+
+-- Migration (forward):
+-- ALTER TABLE orders ADD COLUMN customer_id BIGINT;
+-- CREATE INDEX CONCURRENTLY idx_orders_customer_id ON orders(customer_id);
+
+-- Rollback:
+DROP INDEX CONCURRENTLY IF EXISTS idx_orders_customer_id;
+ALTER TABLE orders DROP COLUMN IF EXISTS customer_id;
+```
+
+## Pre-Migration Checklist
+
+```bash
+# 1. Take snapshot backup
+pgbackrest --stanza=main --type=full backup
+pgbackrest --stanza=main info   # verify backup completed
+
+# 2. Check current lock activity (no long-running transactions)
+psql -c "SELECT pid, now() - query_start AS duration, state, query
+         FROM pg_stat_activity
+         WHERE state != 'idle'
+         ORDER BY duration DESC LIMIT 10;"
+
+# 3. Set statement_timeout for migration session (prevent runaway)
+psql -c "SET statement_timeout = '10min'; <migration_sql>"
+
+# 4. Test rollback on staging first
+```