@pageai/ralph-loop 1.8.0 → 1.10.0

package/.agents/skills/mysql/references/replication-lag.md

@@ -0,0 +1,46 @@
+---
+title: Replication Lag Awareness
+description: Read-replica consistency pitfalls and mitigations
+tags: mysql, replication, lag, read-replicas, consistency, gtid
+---
+
+# Replication Lag
+
+MySQL replication is asynchronous by default. Reads from a replica may return stale data.
+
+## The Core Problem
+1. App writes to primary: `INSERT INTO orders ...`
+2. App immediately reads from replica: `SELECT * FROM orders WHERE id = ?`
+3. Replica hasn't applied the write yet — returns empty or stale data.
+
+## Detecting Lag
+```sql
+-- On the replica
+SHOW REPLICA STATUS\G
+-- Key field: Seconds_Behind_Source (0 = caught up, NULL = not replicating)
+```
+**Warning**: `Seconds_Behind_Source` measures relay-log lag, not true wall-clock staleness. It can underreport during long-running transactions because it only updates when transactions commit.
+
+**GTID-based lag**: for more accurate tracking, compare `@@global.gtid_executed` (replica) to primary GTID position, or use `WAIT_FOR_EXECUTED_GTID_SET()` to wait for a specific transaction.
+
+**Note**: parallel replication with `replica_parallel_type=LOGICAL_CLOCK` requires `binlog_format=ROW`. Statement-based replication (`binlog_format=STATEMENT`) is more limited for parallel apply.
+
+## Mitigation Strategies
+
+| Strategy | How | Trade-off |
+|---|---|---|
+| **Read from primary** | Route critical reads to primary after writes | Increases primary load |
+| **Sticky sessions** | Pin user to primary for N seconds after a write | Adds session affinity complexity |
+| **GTID wait** | `SELECT WAIT_FOR_EXECUTED_GTID_SET('gtid', timeout)` on replica | Adds latency equal to lag |
+| **Semi-sync replication** | Primary waits for >=1 replica ACK before committing | Higher write latency |
+
+## Common Pitfalls
+- **Large transactions cause lag spikes**: A single `INSERT ... SELECT` of 1M rows replays as one big transaction on the replica. Break into batches.
+- **DDL blocks replication**: `ALTER TABLE` with `ALGORITHM=COPY` on primary replays on replica, blocking other relay-log events during execution. `INSTANT` and `INPLACE` DDL are less blocking but still require brief metadata locks.
+- **Long queries on replica**: A slow `SELECT` on the replica can block relay-log application. Use `replica_parallel_workers` (8.0+) with `replica_parallel_type=LOGICAL_CLOCK` for parallel apply. Note: LOGICAL_CLOCK requires `binlog_format=ROW` and `slave_preserve_commit_order=ON` (or `replica_preserve_commit_order=ON`) to preserve commit order.
+- **IO thread bottlenecks**: Network latency, disk I/O, or `relay_log_space_limit` exhaustion can cause lag even when the SQL apply thread isn't saturated. Monitor `Relay_Log_Space` and connectivity.
+
+## Guidelines
+- Assume replicas are always slightly behind. Design reads accordingly.
+- Use GTID-based replication for reliable failover and lag tracking.
+- Monitor `Seconds_Behind_Source` with alerting (>5s warrants investigation).

package/.agents/skills/mysql/references/row-locking-gotchas.md

@@ -0,0 +1,63 @@
+---
+title: InnoDB Row Locking Gotchas
+description: Gap locks, next-key locks, and surprise escalation
+tags: mysql, innodb, locking, gap-locks, next-key-locks, concurrency
+---
+
+# Row Locking Gotchas
+
+InnoDB uses row-level locking, but the actual locked range is often wider than expected.
+
+## Next-Key Locks (REPEATABLE READ)
+InnoDB's default isolation level uses next-key locks for **locking reads** (`SELECT ... FOR UPDATE`, `SELECT ... FOR SHARE`, `UPDATE`, `DELETE`) to prevent phantom reads. A range scan locks every gap in that range. Plain `SELECT` statements use consistent reads (MVCC) and don't acquire locks.
+
+**Exception**: a unique index search with a unique search condition (e.g., `WHERE id = 5` on a unique `id`) locks only the index record, not the gap. Gap/next-key locks still apply for range scans and non-unique searches.
+
+```sql
+-- Locks rows with id 5..10 AND the gaps between them and after the range
+SELECT * FROM orders WHERE id BETWEEN 5 AND 10 FOR UPDATE;
+-- Another session inserting id=7 blocks until the lock is released.
+```
+
+## Gap Locks on Non-Existent Rows
+`SELECT ... FOR UPDATE` on a row that doesn't exist still places a gap lock:
+```sql
+-- No row with id=999 exists, but this locks the gap around where 999 would be
+SELECT * FROM orders WHERE id = 999 FOR UPDATE;
+-- Concurrent INSERTs into that gap are blocked.
+```
+
+## Index-Less UPDATE/DELETE = Full Scan and Broad Locking
+If the WHERE column has no index, InnoDB must scan all rows and locks every row examined (often effectively all rows in the table). This is not table-level locking—InnoDB doesn't escalate locks—but rather row-level locks on all rows:
+```sql
+-- No index on status → locks all rows (not a table lock, but all row locks)
+UPDATE orders SET processed = 1 WHERE status = 'pending';
+-- Fix: CREATE INDEX idx_status ON orders (status);
+```
+
+## SELECT ... FOR SHARE (Shared Locks)
+`SELECT ... FOR SHARE` acquires shared (S) locks instead of exclusive (X) locks. Multiple sessions can hold shared locks simultaneously, but exclusive locks are blocked:
+
+```sql
+-- Session 1: shared lock
+SELECT * FROM orders WHERE id = 5 FOR SHARE;
+
+-- Session 2: also allowed (shared lock)
+SELECT * FROM orders WHERE id = 5 FOR SHARE;
+
+-- Session 3: blocked until shared locks are released
+UPDATE orders SET status = 'processed' WHERE id = 5;
+```
+
+Gap/next-key locks can still apply in REPEATABLE READ, so inserts into locked gaps may be blocked even with shared locks.
+
+## INSERT ... ON DUPLICATE KEY UPDATE
+Takes an exclusive next-key lock on the index entry. If multiple sessions do this concurrently on nearby key values, gap-lock deadlocks are common.
+
+## Lock Escalation Misconception
+InnoDB does **not** automatically escalate row locks to table locks. When a missing index causes "table-wide" locking, it's because InnoDB scans and locks all rows individually—not because locks were escalated.
+
+## Mitigation Strategies
+- **Use READ COMMITTED** when gap locks cause excessive blocking (gap locks disabled in RC except for FK/duplicate-key checks).
+- **Keep transactions short** — hold locks for milliseconds, not seconds.
+- **Ensure WHERE columns are indexed** to avoid full-table lock scans.

package/.agents/skills/postgres/SKILL.md

@@ -0,0 +1,46 @@
+---
+name: postgres
+description: PostgreSQL best practices, query optimization, connection troubleshooting, and performance improvement. Load when working with Postgres databases.
+license: MIT
+metadata:
+  author: planetscale
+  version: "1.0.0"
+---
+
+# PlanetScale Postgres
+
+## Generic Postgres
+
+| Topic                  | Reference                                                        | Use for                                                   |
+| ---------------------- | ---------------------------------------------------------------- | --------------------------------------------------------- |
+| Schema Design          | [references/schema-design.md](https://raw.githubusercontent.com/planetscale/database-skills/main/skills/postgres/references/schema-design.md)           | Tables, primary keys, data types, foreign keys            |
+| Indexing               | [references/indexing.md](https://raw.githubusercontent.com/planetscale/database-skills/main/skills/postgres/references/indexing.md)                      | Index types, composite indexes, performance               |
+| Index Optimization     | [references/index-optimization.md](https://raw.githubusercontent.com/planetscale/database-skills/main/skills/postgres/references/index-optimization.md) | Unused/duplicate index queries, index audit               |
+| Partitioning           | [references/partitioning.md](https://raw.githubusercontent.com/planetscale/database-skills/main/skills/postgres/references/partitioning.md)             | Large tables, time-series, data retention                 |
+| Query Patterns         | [references/query-patterns.md](https://raw.githubusercontent.com/planetscale/database-skills/main/skills/postgres/references/query-patterns.md)         | SQL anti-patterns, JOINs, pagination, batch queries       |
+| Optimization Checklist | [references/optimization-checklist.md](https://raw.githubusercontent.com/planetscale/database-skills/main/skills/postgres/references/optimization-checklist.md) | Pre-optimization audit, cleanup, readiness checks  |
+| MVCC and VACUUM        | [references/mvcc-vacuum.md](https://raw.githubusercontent.com/planetscale/database-skills/main/skills/postgres/references/mvcc-vacuum.md)               | Dead tuples, long transactions, xid wraparound prevention |
+
+## Operations and Architecture
+
+| Topic                  | Reference                                                                    | Use for                                                         |
+| ---------------------- | ---------------------------------------------------------------------------- | --------------------------------------------------------------- |
+| Process Architecture   | [references/process-architecture.md](https://raw.githubusercontent.com/planetscale/database-skills/main/skills/postgres/references/process-architecture.md)     | Multi-process model, connection pooling, auxiliary processes     |
+| Memory Architecture    | [references/memory-management-ops.md](https://raw.githubusercontent.com/planetscale/database-skills/main/skills/postgres/references/memory-management-ops.md)   | Shared/private memory layout, OS page cache, OOM prevention     |
+| MVCC Transactions      | [references/mvcc-transactions.md](https://raw.githubusercontent.com/planetscale/database-skills/main/skills/postgres/references/mvcc-transactions.md)           | Isolation levels, XID wraparound, serialization errors          |
+| WAL and Checkpoints    | [references/wal-operations.md](https://raw.githubusercontent.com/planetscale/database-skills/main/skills/postgres/references/wal-operations.md)                 | WAL internals, checkpoint tuning, durability, crash recovery    |
+| Replication            | [references/replication.md](https://raw.githubusercontent.com/planetscale/database-skills/main/skills/postgres/references/replication.md)                       | Streaming replication, slots, sync commit, failover             |
+| Storage Layout         | [references/storage-layout.md](https://raw.githubusercontent.com/planetscale/database-skills/main/skills/postgres/references/storage-layout.md)                | PGDATA structure, TOAST, fillfactor, tablespaces, disk mgmt     |
+| Monitoring             | [references/monitoring.md](https://raw.githubusercontent.com/planetscale/database-skills/main/skills/postgres/references/monitoring.md)                         | pg_stat views, logging, pg_stat_statements, host metrics        |
+| Backup and Recovery    | [references/backup-recovery.md](https://raw.githubusercontent.com/planetscale/database-skills/main/skills/postgres/references/backup-recovery.md)              | pg_dump, pg_basebackup, PITR, WAL archiving, backup tools      |
+
+## PlanetScale-Specific
+
+| Topic              | Reference                                                                    | Use for                                               |
+| ------------------ | ---------------------------------------------------------------------------- | ----------------------------------------------------- |
+| Connection Pooling | [references/ps-connection-pooling.md](https://raw.githubusercontent.com/planetscale/database-skills/main/skills/postgres/references/ps-connection-pooling.md)   | PgBouncer, pool sizing, pooled vs direct              |
+| Extensions         | [references/ps-extensions.md](https://raw.githubusercontent.com/planetscale/database-skills/main/skills/postgres/references/ps-extensions.md)                   | Supported extensions, compatibility                   |
+| Connections        | [references/ps-connections.md](https://raw.githubusercontent.com/planetscale/database-skills/main/skills/postgres/references/ps-connections.md)                 | Connection troubleshooting, drivers, SSL              |
+| Insights           | [references/ps-insights.md](https://raw.githubusercontent.com/planetscale/database-skills/main/skills/postgres/references/ps-insights.md)                       | Slow queries, MCP server, pscale CLI                  |
+| CLI Commands       | [references/ps-cli-commands.md](https://raw.githubusercontent.com/planetscale/database-skills/main/skills/postgres/references/ps-cli-commands.md)               | pscale CLI reference, branches, deploy requests, auth |
+| CLI API Insights   | [references/ps-cli-api-insights.md](https://raw.githubusercontent.com/planetscale/database-skills/main/skills/postgres/references/ps-cli-api-insights.md)       | Query insights via `pscale api`, schema analysis      |

package/.agents/skills/postgres/references/backup-recovery.md

@@ -0,0 +1,41 @@
+---
+title: Backup and Recovery
+description: Logical/physical backups, PITR, WAL archiving, backup tools, and recovery strategies
+tags: postgres, backup, recovery, pitr, pg_dump, pg_basebackup, wal-archiving, operations
+---
+
+# Backup and Recovery
+
+**FUNDAMENTAL RULE: Backups are useless until you've successfully tested recovery.**
+
+## Logical Backups (pg_dump)
+Exports as SQL or custom format; portable across PG versions and architectures. Formats: `-Fp` (plain SQL), `-Fc` (custom compressed, selective restore), `-Fd` (directory, parallel with `-j`), `-Ft` (tar, avoid). Use `-Fd -j 4` for large DBs. Restore: `pg_restore -d dbname file.dump`; add `-j` for parallel restore. Selective table restore: `pg_restore -t tablename`. Slow for large DBs; RPO = backup frequency (typically 24h).
+
+## Physical Backups (pg_basebackup)
+Copies raw PGDATA; same major version and platform required; cross-architecture works if same endianness (e.g., x86_64 ↔ ARM64). Faster for large clusters; includes all databases. Flags: `-Ft -z -P` for compressed tar with progress. Manual alternative: `pg_backup_start()` → copy PGDATA → `pg_backup_stop()` (complex; must write returned `backup_label`).
+
+## PITR (Point-in-Time Recovery)
+Requires base backup + continuous WAL archiving. Restores to any timestamp, transaction, or named restore point. Without PITR: restore only to backup time (potentially lose hours). With PITR: RPO = minutes. `archive_command` must return 0 ONLY when file is safely stored—premature 0 = data loss risk. `wal_level` must be `replica` or `logical` (not `minimal`).
+
+## WAL Archiving
+`archive_mode=on`, `archive_command='test ! -f /archive/%f && cp %p /archive/%f'`. **Test archive command as postgres user** (not root) since permission issues are common. Monitor `pg_stat_archiver` for `failed_count`, `last_archived_time`. Archive failures prevent WAL recycling → disk fills.
+
+## Tool Comparison
+| Tool | Use case |
+|------|----------|
+| pg_dump | Small DBs, migrations, selective restore |
+| pg_basebackup | Basic PITR, built-in |
+| pgBackRest | Production—parallel, incremental, S3/GCS/Azure, retention |
+| Barman | Enterprise PITR, retention policies |
+| WAL-G | Cloud-native, S3/GCS/Azure |
+
+## RPO/RTO
+Logical only: RPO = backup interval (hours); RTO = hours. PITR: RPO = minutes; RTO = hours. Synchronous replication: RPO = 0; RTO = seconds to minutes (failover).
+
+## Operational Rules
+- Verify integrity with `pg_verifybackup` (PG 13+)
+- Test recovery / PITR regularly
+- Take backups from standby to avoid impacting primary
+- Retention: 7 daily, 4 weekly, 12 monthly
+- Monitor archive growth and backup age
+- **Never assume backups work without testing**

package/.agents/skills/postgres/references/index-optimization.md

@@ -0,0 +1,69 @@
+---
+title: Index Optimization Queries
+description: Index audit queries
+tags: postgres, indexes, unused-indexes, duplicate-indexes, optimization
+---
+
+# Index Optimization
+
+## Identify Unused Indexes
+
+Query to find unused indexes:
+
+```sql
+-- indexes with 0 scans (check pg_stat_reset / pg_postmaster_start_time first)
+SELECT
+   s.schemaname,
+   s.relname AS table_name,
+   s.indexrelname AS index_name,
+   pg_size_pretty(pg_relation_size(s.indexrelid)) AS index_size
+ FROM pg_catalog.pg_stat_user_indexes s
+ JOIN pg_catalog.pg_index i ON s.indexrelid = i.indexrelid
+ WHERE s.idx_scan = 0
+   AND 0 <> ALL (i.indkey)       -- exclude expression indexes
+   AND NOT i.indisunique          -- exclude UNIQUE indexes
+   AND NOT EXISTS (               -- exclude constraint-backing indexes
+     SELECT 1 FROM pg_catalog.pg_constraint c
+     WHERE c.conindid = s.indexrelid
+   )
+ ORDER BY pg_relation_size(s.indexrelid) DESC;
+```
+
+## Indexes Per Table Guidelines
+
+- **< 5**: Normal
+- **5-10**: Monitor (Verify necessity)
+- **> 10**: Audit required (High write overhead)
+
+```sql
+SELECT relname AS table, count(*) as index_count
+FROM pg_stat_user_indexes
+GROUP BY relname
+ORDER BY count(*) DESC;
+```
+
+## Identify Unused Indexes
+
+Indexes with identical definitions (after normalizing names) on the same table are duplicates:
+
+```sql
+SELECT
+  schemaname || '.' || tablename AS table,
+  array_agg(indexname) AS duplicate_indexes,
+  pg_size_pretty(sum(pg_relation_size((schemaname || '.' || indexname)::regclass))) AS total_size
+FROM pg_indexes
+WHERE schemaname NOT IN ('pg_catalog', 'information_schema')
+GROUP BY schemaname, tablename,
+  regexp_replace(indexdef, 'INDEX \S+ ON ', 'INDEX ON ')
+HAVING count(*) > 1;
+```
+
+**Always confirm with a human before dropping or removing any indexes identified by the queries above.** Even indexes with 0 scans may be needed for infrequent but critical queries, and stats may have been reset recently.
+
+## Per-table Index Count Guidelines
+
+| Index Count | Recommendation                              |
+| ----------- | ------------------------------------------- |
+| <5          | Normal                                      |
+| 5-10        | Review for unused/duplicates                |
+| >10         | Audit required - significant write overhead |

package/.agents/skills/postgres/references/indexing.md

@@ -0,0 +1,61 @@
+---
+title: Indexing Best Practices
+description: Index design guide
+tags: postgres, indexes, composite, partial, covering, gin, brin
+---
+
+# Indexing Best Practices
+
+## Core Rules
+
+1. **Always index foreign key columns** — PostgreSQL does not auto-create these
+2. **Index columns in WHERE, JOIN, and ORDER BY** clauses
+3. **Don't over-index** — each index slows writes and uses storage
+4. **Verify with EXPLAIN ANALYZE** — confirm indexes are actually used
+
+## Composite Indexes
+
+Put equality columns first, then range/sort columns:
+
+```sql
+-- WHERE status = 'active' AND created_at > '2026-01-01'
+CREATE INDEX order_status_created_idx ON order (status, created_at);
+```
+
+A composite index on `(a, b)` supports queries on `a` + `b` and `a` alone, but not `b` alone.
+
+## Partial Indexes
+
+Reduce index size by filtering to common query patterns.
+Only use if index size is problematic but the index is needed for performance.
+
+```sql
+CREATE INDEX order_active_idx ON order (customer_id)
+  WHERE status = 'active';
+```
+
+## Covering Indexes
+
+Consider creating covering indexes for commonly executed query patterns that return only 1 or a small number of columns.
+
+## Index Types
+
+| Type | Use Case | Example |
+| --- | --- | --- |
+| B-tree (default) | Equality, range, sorting | `WHERE id = 1`, `ORDER BY date` |
+| GIN | Arrays, JSONB, full-text | `WHERE tags @> ARRAY['x']` |
+| GiST | Geometric, range types, full-text | PostGIS, `tsrange`, `tsvector` |
+| BRIN | Large sequential/time-series | Append-only logs, events (requires physical row order correlation) |
+
+```sql
+CREATE INDEX metadata_idx ON order USING GIN (metadata);       -- JSONB
+CREATE INDEX event_created_idx ON event USING BRIN (created_at); -- time-series
+```
+
+## Guidelines
+
+- Name indexes consistently: `{table}_{column}_idx`
+- Review for unused indexes periodically
+- **Always confirm with a human before removing or dropping any indexes** — even unused ones may serve a purpose not reflected in recent stats
+- Use partial indexes for frequently filtered subsets
+- Use covering indexes on hot read paths

package/.agents/skills/postgres/references/memory-management-ops.md

@@ -0,0 +1,39 @@
+---
+title: Memory Architecture and OOM Prevention
+description: PostgreSQL shared/private memory layout, OS page cache interaction, and OOM avoidance strategies
+tags: postgres, memory, shared_buffers, work_mem, oom, architecture, operations
+---
+
+# Memory Architecture and OOM Prevention
+
+## Memory Areas
+
+- **Shared memory**: `shared_buffers` — main data cache, all processes, requires restart to change.
+- **Private per backend**: `work_mem` (sorts/hashes/joins, per-operation); `maintenance_work_mem` (VACUUM, CREATE INDEX, ALTER TABLE ADD FOREIGN KEY); `temp_buffers` (8MB default).
+- **Planner hint only**: `effective_cache_size` is NOT allocated — set to ~50–75% of total RAM.
+- **Hash multiplier**: `hash_mem_multiplier` (default 2.0) means hash ops use up to 2× `work_mem`.
+
+## Memory Multiplication Danger
+
+Maximum potential: `work_mem × operations_per_query × (parallel_workers + 1) × connections` (leader participates by default via `parallel_leader_participation = on`; hash operations use up to `hash_mem_multiplier × work_mem`, default 2.0). Example: 128MB work_mem, 3 ops (2 sorts + 1 hash join), 2 parallel workers, 100 connections → 2 sorts at 128MB = 256MB, 1 hash join at 128MB × 2.0 = 256MB, per process = 512MB, × 3 processes (2 workers + leader) = 1536MB/query, × 100 connections = **~150GB** worst case. This case is rare.
+Not all queries hit limits at once, but high concurrency + large datasets approach it. This is a common cause of OOM in containerized/Kubernetes deployments. Plan capacity with a 1.5–2× safety margin.
+
+## OS Page Cache (Double Buffering)
+
+Data exists in both `shared_buffers` and OS page cache. A miss in shared_buffers can still hit OS cache (avoiding disk I/O). Extremely large shared_buffers can hurt performance: less OS cache, slower startup, heavier checkpoints. Optimal split depends on workload (OLTP vs OLAP).
+
+## OOM Prevention
+
+- Implement connection pooling to reduce total backend count.
+- Reduce `work_mem` globally; use per-session overrides for heavy queries only.
+- Lower `max_parallel_workers_per_gather` in high-concurrency systems.
+- Set `statement_timeout` to kill runaway queries.
+- Monitor: `dmesg -T | grep "killed process"` and `temp_blks_written` in pg_stat_statements.
+
+## Operational Rules
+
+- Tune per-session first, global last.
+- Suspect OOM when memory spikes during high concurrency, dashboards, or large batch jobs.
+- Increase memory only after confirming spill behavior (`temp_blks_written > 0`).
+- `maintenance_work_mem` can be set much higher (1–2GB) — fewer processes use it. Cap autovacuum with `autovacuum_work_mem` to avoid `autovacuum_max_workers × maintenance_work_mem` memory spikes.
+- `shared_buffers` change requires full restart; `work_mem` is per-session changeable.

package/.agents/skills/postgres/references/monitoring.md

@@ -0,0 +1,59 @@
+---
+title: Monitoring
+description: Essential PostgreSQL monitoring views, pg_stat_statements, logging, host metrics, and statistics management
+tags: postgres, monitoring, pg_stat_statements, logging, pgbadger, metrics, operations
+---
+
+# Monitoring
+
+## Essential Views
+
+- **pg_stat_activity**: First stop when something is wrong — running queries, states, wait events, locks.
+- **pg_stat_statements**: Execution stats for all SQL. Requires `shared_preload_libraries = 'pg_stat_statements'` and `CREATE EXTENSION pg_stat_statements`.
+- **pg_stat_database**: Cache hit ratio, temp files, deadlocks, connections per database.
+- **pg_stat_user_tables**: `seq_scan` vs `idx_scan`, dead tuples, last vacuum/analyze times.
+- **pg_stat_user_indexes**: Find unused indexes (`idx_scan = 0` with large size).
+- **pg_stat_bgwriter**: `buffers_clean`, `maxwritten_clean`, `buffers_alloc`. Pre-PG 17 also had `buffers_checkpoint`, `buffers_backend` (high = backends bypassing bgwriter). PG 17+ moved checkpoint stats to `pg_stat_checkpointer`.
+- **pg_stat_checkpointer** (PG 17+): Checkpoint frequency (`num_timed`, `num_requested`), write/sync time.
+
+## Key Queries
+
+```sql
+-- Slow queries (with cache hit ratio)
+SELECT query, calls, mean_exec_time,
+  100.0 * shared_blks_hit / nullif(shared_blks_hit + shared_blks_read, 0) AS cache_hit_pct
+FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 10;
+
+-- Connection counts / states
+SELECT state, count(*) FROM pg_stat_activity GROUP BY state;
+
+-- Dead tuples (vacuum candidates)
+SELECT relname, n_dead_tup, last_autovacuum FROM pg_stat_user_tables ORDER BY n_dead_tup DESC;
+-- last_autovacuum = <null> means autovacuum has not run on this table
+```
+
+Blocking: use `pg_blocking_pids(pid)` with `pg_stat_activity` to find blocked and blocking sessions.
+
+## Logging — First Line of Defense
+
+PostgreSQL is extremely vocal about problems. **Always check logs first**: `tail -f /var/log/postgresql/postgresql-*.log`.
+
+Key settings: `log_min_duration_statement` (OLTP: 1–3s, analytics: 30–60s, dev: 100–500ms). Enable `log_checkpoints=on`, `log_connections=on`, `log_disconnections=on`, `log_lock_waits=on`, `log_temp_files=0`. Use CSV log format for pgBadger analysis; pgBadger generates HTML reports with query stats and performance graphs.
+
+## pg_activity
+
+Interactive top-like tool (pip install pg_activity). Run on DB host for OS metrics alongside PG metrics. Combines `pg_stat_activity` with CPU/memory/I/O context.
+
+## Host Metrics — Critical
+
+PostgreSQL cannot report these. **Monitor them yourself:**
+
+- **CPU**: Steal time >10% in VMs bad; load average > core count; context switches >100k/sec.
+- **Memory**: Any swap = performance degradation. Check `dmesg` for OOM kills.
+- **Disk I/O**: `iostat -x` — `%util=100%` means saturated; `await` >10ms = high latency.
+- **Disk space**: >90% critical (VACUUM fails, writes fail). Check inode usage too.
+- **Network**: Packet loss >0% = problems; high retransmits = instability.
+
+## Statistics Management
+
+Stats accumulate since last reset or restart; check `stats_reset` timestamp. `pg_stat_statements_reset()` clears query stats; `pg_stat_reset()` clears database stats. Reset after major maintenance, config changes, or perf testing — not routinely. Prefer snapshotting stats to external monitoring (Prometheus, Datadog) over resetting. **Always confirm with a human before resetting statistics** — resetting destroys historical performance baselines and can make it harder to identify unused indexes or regressions.

package/.agents/skills/postgres/references/mvcc-transactions.md

@@ -0,0 +1,38 @@
+---
+title: MVCC Transactions and Concurrency
+description: Transaction isolation levels, XID wraparound prevention, serialization errors, and long-transaction impact
+tags: postgres, mvcc, transactions, isolation, xid-wraparound, concurrency, serialization
+---
+
+# MVCC Transactions and Concurrency
+
+## Transaction Isolation Levels
+
+- **READ UNCOMMITTED** — treated as READ COMMITTED in PostgreSQL; no dirty reads ever.
+- **READ COMMITTED** (default): new snapshot per statement; can see different data within same tx.
+- **REPEATABLE READ**: snapshot at first query; can cause serialization errors on write conflicts.
+- **SERIALIZABLE**: strongest; transactions appear serial; requires retry logic in app code.
+
+Readers never block writers; writers never block readers (only writer-writer conflicts on same row). No lock escalation — row locks never degrade to table locks.
+
+## XID Wraparound
+
+32-bit transaction IDs wrap at ~2 billion (2^31). `VACUUM FREEZE` replaces old XIDs with FrozenXID (value 2, always visible). Without freeze: after wraparound, old rows appear "in the future" and become **invisible**. Data physically exists but is invisible to all queries — looks like total data loss. PostgreSQL emergency shutdown at 2B XIDs to prevent this. XID wraparound should be avoided at all cost.
+
+Warning messages start at ~1.4B XIDs; shutdown at 2B. Recovery requires single-user mode VACUUM — can take hours to days on large DBs. **Never disable autovacuum** — it's your protection against wraparound.
+
+## XID Age Monitoring
+
+```sql
+SELECT datname, age(datfrozenxid),
+  ROUND(100.0 * age(datfrozenxid) / 2147483648, 2) AS pct
+FROM pg_database ORDER BY age(datfrozenxid) DESC;
+```
+
+## Long Transaction Impact
+
+A single long-running transaction blocks VACUUM from removing dead tuples across the **entire database**. Causes table bloat, increased disk, slower queries, cache pollution. `idle_in_transaction` connections are the #1 operational MVCC issue. Set `idle_in_transaction_session_timeout` (30s–5min). Dead tuples waste I/O on seq scans and cause useless heap lookups from indexes.
+
+## Serialization Errors
+
+Apps **must** handle "could not serialize access" with retry logic. More common in REPEATABLE READ and SERIALIZABLE. Smaller, faster transactions reduce conflict frequency.

package/.agents/skills/postgres/references/mvcc-vacuum.md

@@ -0,0 +1,41 @@
+---
+title: MVCC and VACUUM
+description: MVCC internals, VACUUM/autovacuum tuning, and bloat prevention
+tags: postgres, mvcc, vacuum, autovacuum, xid, bloat, dead-tuples
+---
+
+# MVCC and VACUUM
+
+## MVCC
+
+Every `UPDATE` creates a new tuple and marks the old one dead; `DELETE` marks tuples dead. Dead tuples accumulate until `VACUUM` reclaims space. Each transaction gets a 32-bit XID (2^32 ≈ 4B values, but modular comparison means the effective danger zone is 2^31 ≈ 2B). VACUUM must freeze old XIDs to prevent wraparound.
+
+## VACUUM vs VACUUM FULL
+
+`VACUUM` is non-blocking (ShareUpdateExclusive lock) and marks dead space reusable. `VACUUM FULL` rewrites the table and requires an AccessExclusive lock — use only as a last resort. For online bloat reduction prefer `pg_squeeze` or `pg_repack`.
+
+## Autovacuum Tuning
+
+Triggers when dead tuples > `Min(autovacuum_vacuum_max_threshold, autovacuum_vacuum_threshold + autovacuum_vacuum_scale_factor * reltuples)`. `autovacuum_vacuum_max_threshold` defaults to 100M (PG 18+), capping the threshold for very large tables. Also triggers on inserts exceeding `autovacuum_vacuum_insert_threshold + autovacuum_vacuum_insert_scale_factor * reltuples * pct_not_frozen` (ensures insert-only tables get frozen; PG 13+). For large/hot tables, set per-table overrides:
+
+- `autovacuum_vacuum_scale_factor` — default 0.2; lower to 0.01–0.05 for large tables.
+- `autovacuum_vacuum_cost_delay` — default 2 ms; set to 0 on fast storage.
+- `autovacuum_vacuum_cost_limit` — default -1 (uses `vacuum_cost_limit`, effectively 200); raise to 1000–2000 on fast storage.
+- `autovacuum_freeze_max_age` — default 200M; triggers anti-wraparound vacuum.
+- `vacuum_failsafe_age` — default 1.6B; last-resort mode (PG 14+) that disables throttling and skips index vacuuming when wraparound is imminent.
+
+## Key Monitoring Queries
+
+Dead tuples: `SELECT relname, n_dead_tup, last_autovacuum FROM pg_stat_user_tables ORDER BY n_dead_tup DESC;`
+
+XID age: `SELECT datname, age(datfrozenxid) AS xid_age FROM pg_database ORDER BY xid_age DESC;`
+
+Long transactions: `SELECT pid, state, now() - xact_start AS tx_age FROM pg_stat_activity WHERE xact_start IS NOT NULL ORDER BY xact_start;`
+
+## Best Practices
+
+- Keep transactions short; set `idle_in_transaction_session_timeout` (30s–5min).
+- Alert when `age(datfrozenxid)` exceeds 40–50% of wraparound (~800M–1B).
+- Tune autovacuum per-table for write-heavy tables; don't change global defaults first.
+- Fix application transaction scope before adjusting vacuum parameters.
+- Never disable autovacuum globally.

package/.agents/skills/postgres/references/optimization-checklist.md

@@ -0,0 +1,19 @@
+---
+title: Database Optimization Checklist
+description: Optimize checklist
+tags: postgres, optimization, indexes, partitioning, maintenance
+---
+
+# Optimization Checklist
+
+When optimizing performance, check the following:
+
+- Look for unused indexes (0 scans; exclude unique/primary indexes and verify stats age first)
+- Look for duplicate indexes
+- Archive audit/log tables >10GB
+- Review tables >500GB for partitioning (>100GB for time-series/logs)
+- Verify all extensions are supported
+- Check for circular foreign key dependencies
+- Consider alternatives to UUID primary keys for large tables
+- Configure connection pooling for OLTP workloads
+- **Always confirm with a human before removing any indexes, dropping partitions, archiving tables, or performing other destructive actions**

package/.agents/skills/postgres/references/partitioning.md

@@ -0,0 +1,79 @@
+---
+title: Table Partitioning Guide
+description: Partition guide
+tags: postgres, partitioning, range, list, pg_partman, data-retention
+---
+
+# Table Partitioning
+
+Plan partitioning upfront for tables expected to grow large. Retrofitting later requires a migration.
+
+## When to Partition
+
+Partitioning benefits maintenance (vacuum, index builds) and data retention more than pure query speed.
+
+| Table Type | Size Threshold | Row Threshold |
+| --- | --- | --- |
+| General tables | >100 GB (or >RAM) | >20M rows |
+| Time-series / logs | >50 GB | >10M rows |
+
+Use the lower thresholds for append-heavy, time-ordered data with retention needs (logs, events, audit trails, metrics).
+
+## Range Partitioning (Most Common)
+
+```sql
+-- EXAMPLE
+CREATE TABLE event (
+  id BIGINT GENERATED ALWAYS AS IDENTITY,
+  event_type TEXT NOT NULL,
+  payload JSONB,
+  created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+  PRIMARY KEY (id, created_at) -- Partition key MUST be part of PK
+) PARTITION BY RANGE (created_at);
+
+CREATE TABLE event_2026_01 PARTITION OF event
+  FOR VALUES FROM ('2026-01-01') TO ('2026-02-01');
+
+CREATE TABLE event_2026_02 PARTITION OF event
+  FOR VALUES FROM ('2026-02-01') TO ('2026-03-01');
+```
+
+## List Partitioning
+
+Useful for partitioning by region, tenant, or category:
+
+```sql
+-- EXAMPLE
+CREATE TABLE order (
+  id BIGINT GENERATED ALWAYS AS IDENTITY,
+  region TEXT NOT NULL,
+  total NUMERIC(10,2),
+  PRIMARY KEY (id, region) -- Partition key MUST be part of PK
+) PARTITION BY LIST (region);
+
+CREATE TABLE order_us PARTITION OF order FOR VALUES IN ('us');
+CREATE TABLE order_eu PARTITION OF order FOR VALUES IN ('eu');
+CREATE TABLE order_default PARTITION OF order DEFAULT;  -- catches unmatched values
+```
+
+## Partition Management
+
+- Use `pg_partman` (extension) to automate partition creation and cleanup.
+- Use `DETACH PARTITION` to remove a partition while retaining it as a standalone table (e.g., for archiving).
+- Use `DETACH PARTITION ... CONCURRENTLY` (PG 14+) to avoid `ACCESS EXCLUSIVE` locks on the parent table.
+- Drop old partitions for data retention instead of `DELETE` to avoid vacuum overhead and bloat.
+- Create future partitions ahead of time to avoid insert failures.
+- **Always confirm with a human before detaching or dropping partitions.** These are destructive actions — detaching removes data from the partitioned table, and dropping permanently deletes the data.
+
+```sql
+-- DESTRUCTIVE: confirm with a human before executing
+ALTER TABLE event DETACH PARTITION event_2025_01 CONCURRENTLY;
+DROP TABLE event_2025_01;
+```
+
+## Guidelines & Limitations
+
+- **Primary Keys**: Partition key columns MUST be included in the `PRIMARY KEY` and any `UNIQUE` constraints.
+- **Global Uniqueness**: Global unique constraints on non-partition columns are NOT supported.
+- **Indexes**: Indexes defined on the parent are automatically created on all partitions (and future ones).
+- **Pruning**: Ensure queries filter by the partition key to enable "partition pruning" (skipping unrelated partitions).