aios-core 2.1.6 → 2.2.0

package/.aios-core/product/data/postgres-tuning-guide.md

@@ -0,0 +1,300 @@
+# PostgreSQL Performance Tuning Guide
+
+**Purpose:** Reference guide for PostgreSQL performance optimization
+**Agent:** Dan (Data Engineer)
+**Standard:** Production-ready PostgreSQL configurations
+
+---
+
+## CONFIGURATION TUNING
+
+### Memory Settings
+
+#### shared_buffers
+- **Purpose:** Shared memory for caching data
+- **Recommendation:** 25% of total RAM (max ~8GB for most workloads)
+```sql
+-- Check current value
+SHOW shared_buffers;
+
+-- Example: 8GB RAM system
+-- Set to 2GB (in postgresql.conf)
+shared_buffers = 2GB
+```
+
+#### effective_cache_size
+- **Purpose:** Planner's estimate of available cache
+- **Recommendation:** 50-75% of total RAM
+```sql
+-- Example: 8GB RAM system
+effective_cache_size = 6GB
+```
+
+#### work_mem
+- **Purpose:** Memory per operation (sort, hash)
+- **Recommendation:** total_ram / max_connections / 4
+- **Caution:** Set too high can cause memory exhaustion
+```sql
+-- Example: 8GB RAM, 100 connections
+work_mem = 20MB
+
+-- For specific queries needing more
+SET work_mem = '256MB';
+-- Run query
+RESET work_mem;
+```
+
+#### maintenance_work_mem
+- **Purpose:** Memory for maintenance operations (VACUUM, CREATE INDEX)
+- **Recommendation:** 256MB-1GB depending on RAM
+```sql
+maintenance_work_mem = 512MB
+```
+
+---
+
+## CONNECTION POOLING
+
+### Why Pool Connections
+- PostgreSQL forks a process per connection (~10MB each)
+- Too many connections = memory exhaustion
+- Connection overhead is significant
+
+### PgBouncer Configuration
+```ini
+[databases]
+mydb = host=localhost port=5432 dbname=mydb
+
+[pgbouncer]
+listen_port = 6432
+listen_addr = *
+auth_type = md5
+auth_file = /etc/pgbouncer/userlist.txt
+pool_mode = transaction
+max_client_conn = 1000
+default_pool_size = 20
+min_pool_size = 5
+reserve_pool_size = 5
+```
+
+### Pool Modes
+- **session:** Connection held until client disconnects
+- **transaction:** Connection returned after transaction (recommended)
+- **statement:** Connection returned after each statement
+
+### Supabase Connection Pooling
+- Built-in Supavisor pooler
+- Use pooler URL for application connections
+- Use direct URL for migrations only
+
+---
+
+## QUERY OPTIMIZATION
+
+### EXPLAIN ANALYZE
+```sql
+EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
+SELECT * FROM orders
+WHERE customer_id = 123
+ORDER BY created_at DESC
+LIMIT 10;
+```
+
+### Key Metrics to Watch
+- **Seq Scan:** Full table scan (may indicate missing index)
+- **Rows Removed by Filter:** High count = inefficient query
+- **Buffers:** Shared hit (cache) vs read (disk)
+- **Actual Time:** Startup time vs total time
+
+### Common Optimizations
+
+#### Add Missing Indexes
+```sql
+-- Before: Seq Scan on orders
+EXPLAIN SELECT * FROM orders WHERE customer_id = 123;
+
+-- Add index
+CREATE INDEX idx_orders_customer_id ON orders(customer_id);
+
+-- After: Index Scan on idx_orders_customer_id
+```
+
+#### Use Covering Indexes
+```sql
+-- Query
+SELECT email, name FROM users WHERE email = 'test@example.com';
+
+-- Covering index (includes all columns needed)
+CREATE INDEX idx_users_email_covering ON users(email) INCLUDE (name);
+```
+
+#### Partial Indexes
+```sql
+-- Only index active users
+CREATE INDEX idx_users_active ON users(email)
+WHERE is_active = true;
+```
+
+---
+
+## VACUUM AND MAINTENANCE
+
+### Autovacuum Tuning
+```sql
+-- Check autovacuum stats
+SELECT schemaname, relname, n_dead_tup, last_autovacuum
+FROM pg_stat_user_tables
+ORDER BY n_dead_tup DESC;
+
+-- Per-table settings for high-churn tables
+ALTER TABLE high_churn_table SET (
+  autovacuum_vacuum_scale_factor = 0.1,
+  autovacuum_analyze_scale_factor = 0.05
+);
+```
+
+### Manual Maintenance
+```sql
+-- Analyze table statistics
+ANALYZE table_name;
+
+-- Vacuum (reclaim space)
+VACUUM table_name;
+
+-- Vacuum + analyze
+VACUUM ANALYZE table_name;
+
+-- Full vacuum (locks table, rewrites)
+VACUUM FULL table_name;  -- Use with caution
+```
+
+### Reindex
+```sql
+-- Rebuild bloated index (non-blocking)
+REINDEX INDEX CONCURRENTLY idx_name;
+
+-- Rebuild all indexes on table
+REINDEX TABLE CONCURRENTLY table_name;
+```
+
+---
+
+## MONITORING QUERIES
+
+### Find Slow Queries
+```sql
+-- Enable pg_stat_statements
+CREATE EXTENSION IF NOT EXISTS pg_stat_statements;
+
+-- Top 10 slowest queries
+SELECT
+  calls,
+  round(total_exec_time::numeric, 2) as total_ms,
+  round(mean_exec_time::numeric, 2) as avg_ms,
+  query
+FROM pg_stat_statements
+ORDER BY mean_exec_time DESC
+LIMIT 10;
+```
+
+### Check Index Usage
+```sql
+-- Unused indexes
+SELECT
+  schemaname,
+  relname,
+  indexrelname,
+  idx_scan,
+  pg_size_pretty(pg_relation_size(indexrelid)) as size
+FROM pg_stat_user_indexes
+WHERE idx_scan = 0
+AND schemaname NOT IN ('pg_catalog', 'pg_toast')
+ORDER BY pg_relation_size(indexrelid) DESC;
+```
+
+### Table Bloat
+```sql
+-- Check table sizes and bloat
+SELECT
+  schemaname,
+  relname,
+  n_live_tup,
+  n_dead_tup,
+  round(100 * n_dead_tup / NULLIF(n_live_tup + n_dead_tup, 0), 2) as dead_pct
+FROM pg_stat_user_tables
+WHERE n_dead_tup > 0
+ORDER BY n_dead_tup DESC;
+```
+
+### Cache Hit Ratio
+```sql
+-- Should be > 99% for good performance
+SELECT
+  round(100 * sum(blks_hit) / sum(blks_hit + blks_read), 2) as cache_hit_ratio
+FROM pg_stat_database;
+```
+
+---
+
+## LOCKING AND CONCURRENCY
+
+### Check Active Locks
+```sql
+SELECT
+  l.pid,
+  l.mode,
+  l.granted,
+  a.usename,
+  a.query,
+  a.state
+FROM pg_locks l
+JOIN pg_stat_activity a ON l.pid = a.pid
+WHERE NOT l.granted;
+```
+
+### Kill Long-Running Queries
+```sql
+-- Find long-running queries
+SELECT
+  pid,
+  now() - pg_stat_activity.query_start AS duration,
+  query,
+  state
+FROM pg_stat_activity
+WHERE state != 'idle'
+AND now() - pg_stat_activity.query_start > interval '5 minutes';
+
+-- Cancel query (graceful)
+SELECT pg_cancel_backend(pid);
+
+-- Terminate connection (force)
+SELECT pg_terminate_backend(pid);
+```
+
+---
+
+## PRODUCTION CHECKLIST
+
+### Before Go-Live
+- [ ] shared_buffers configured (25% RAM)
+- [ ] effective_cache_size configured (50-75% RAM)
+- [ ] work_mem tuned for workload
+- [ ] Connection pooling configured
+- [ ] Autovacuum tuned for high-churn tables
+- [ ] pg_stat_statements enabled
+- [ ] Slow query logging enabled
+- [ ] Backup strategy tested
+- [ ] Index strategy reviewed
+
+### Regular Maintenance
+- [ ] Monitor cache hit ratio (>99%)
+- [ ] Check unused indexes monthly
+- [ ] Review slow query logs weekly
+- [ ] Analyze table statistics after bulk loads
+- [ ] Monitor table bloat
+- [ ] Test backup restoration quarterly
+
+---
+
+**Reviewer:** ________ **Date:** ________
+**Quality Gate:** [ ] PASS [ ] NEEDS REVIEW

package/.aios-core/product/data/rls-security-patterns.md

@@ -0,0 +1,333 @@
+# Row Level Security (RLS) Patterns Guide
+
+**Purpose:** Reference guide for implementing secure RLS policies
+**Agent:** Dan (Data Engineer)
+**Platform:** PostgreSQL / Supabase
+**Security:** Multi-tenant data isolation patterns
+
+---
+
+## RLS FUNDAMENTALS
+
+### Enabling RLS
+```sql
+-- Enable RLS on table (required before policies work)
+ALTER TABLE posts ENABLE ROW LEVEL SECURITY;
+
+-- Force RLS for table owner (recommended in Supabase)
+ALTER TABLE posts FORCE ROW LEVEL SECURITY;
+```
+
+### Policy Structure
+```sql
+CREATE POLICY policy_name
+ON table_name
+FOR operation           -- ALL, SELECT, INSERT, UPDATE, DELETE
+TO role                 -- PUBLIC, authenticated, specific_role
+USING (expression)      -- Filter for SELECT, UPDATE, DELETE
+WITH CHECK (expression) -- Filter for INSERT, UPDATE
+```
+
+---
+
+## COMMON PATTERNS
+
+### Pattern 1: User Owns Row
+```sql
+-- Users can only see/modify their own data
+CREATE POLICY "Users can view own data"
+ON profiles FOR SELECT
+TO authenticated
+USING (auth.uid() = user_id);
+
+CREATE POLICY "Users can update own data"
+ON profiles FOR UPDATE
+TO authenticated
+USING (auth.uid() = user_id)
+WITH CHECK (auth.uid() = user_id);
+
+CREATE POLICY "Users can delete own data"
+ON profiles FOR DELETE
+TO authenticated
+USING (auth.uid() = user_id);
+
+CREATE POLICY "Users can insert own data"
+ON profiles FOR INSERT
+TO authenticated
+WITH CHECK (auth.uid() = user_id);
+```
+
+### Pattern 2: Organization/Team Based
+```sql
+-- Users can see data from their organization
+CREATE POLICY "Team members can view team data"
+ON projects FOR SELECT
+TO authenticated
+USING (
+  organization_id IN (
+    SELECT organization_id
+    FROM organization_members
+    WHERE user_id = auth.uid()
+  )
+);
+```
+
+### Pattern 3: Role-Based Access
+```sql
+-- Different access levels based on user role
+CREATE POLICY "Admins have full access"
+ON sensitive_data FOR ALL
+TO authenticated
+USING (
+  EXISTS (
+    SELECT 1 FROM user_roles
+    WHERE user_id = auth.uid()
+    AND role = 'admin'
+  )
+);
+
+CREATE POLICY "Regular users read-only"
+ON sensitive_data FOR SELECT
+TO authenticated
+USING (
+  EXISTS (
+    SELECT 1 FROM user_roles
+    WHERE user_id = auth.uid()
+    AND role = 'user'
+  )
+);
+```
+
+### Pattern 4: Public Read, Authenticated Write
+```sql
+-- Anyone can read, only authenticated can write
+CREATE POLICY "Public read access"
+ON public_content FOR SELECT
+TO PUBLIC
+USING (true);
+
+CREATE POLICY "Authenticated write access"
+ON public_content FOR INSERT
+TO authenticated
+WITH CHECK (auth.uid() = author_id);
+```
+
+### Pattern 5: Time-Based Access
+```sql
+-- Access expires after a certain date
+CREATE POLICY "Time-limited access"
+ON trial_content FOR SELECT
+TO authenticated
+USING (
+  trial_expires_at > now()
+  AND user_id = auth.uid()
+);
+```
+
+---
+
+## SUPABASE-SPECIFIC PATTERNS
+
+### Using auth.uid()
+```sql
+-- Get the current authenticated user's ID
+SELECT auth.uid();
+
+-- In policy
+CREATE POLICY "Owner access"
+ON documents FOR ALL
+TO authenticated
+USING (owner_id = auth.uid());
+```
+
+### Using auth.jwt()
+```sql
+-- Access JWT claims
+SELECT auth.jwt() ->> 'email';
+SELECT auth.jwt() -> 'app_metadata' ->> 'role';
+
+-- Policy using custom claims
+CREATE POLICY "Premium users only"
+ON premium_content FOR SELECT
+TO authenticated
+USING (
+  (auth.jwt() -> 'app_metadata' ->> 'plan') = 'premium'
+);
+```
+
+### Using auth.role()
+```sql
+-- Different policies for different Supabase roles
+CREATE POLICY "Anon can read public"
+ON content FOR SELECT
+TO anon
+USING (is_public = true);
+
+CREATE POLICY "Authenticated can read all"
+ON content FOR SELECT
+TO authenticated
+USING (true);
+
+CREATE POLICY "Service role bypasses RLS"
+ON content FOR ALL
+TO service_role
+USING (true);
+```
+
+---
+
+## PERFORMANCE OPTIMIZATION
+
+### Use Indexes for RLS
+```sql
+-- Create index on columns used in RLS policies
+CREATE INDEX idx_posts_user_id ON posts(user_id);
+CREATE INDEX idx_org_members_user_org ON organization_members(user_id, organization_id);
+```
+
+### Avoid Expensive Subqueries
+```sql
+-- ❌ Bad: Subquery in every row check
+CREATE POLICY "Expensive policy"
+ON documents FOR SELECT
+USING (
+  owner_id IN (
+    SELECT user_id FROM complex_permissions_view
+    WHERE /* complex logic */
+  )
+);
+
+-- ✅ Better: Use a security definer function
+CREATE OR REPLACE FUNCTION get_accessible_document_ids()
+RETURNS SETOF uuid
+LANGUAGE sql
+SECURITY DEFINER
+STABLE
+AS $$
+  SELECT document_id FROM user_document_access
+  WHERE user_id = auth.uid()
+$$;
+
+CREATE POLICY "Optimized policy"
+ON documents FOR SELECT
+USING (id IN (SELECT get_accessible_document_ids()));
+```
+
+### Materialized Permissions
+```sql
+-- Pre-compute permissions for complex access patterns
+CREATE TABLE user_document_access (
+  user_id uuid REFERENCES auth.users,
+  document_id uuid REFERENCES documents,
+  PRIMARY KEY (user_id, document_id)
+);
+
+CREATE INDEX idx_uda_user ON user_document_access(user_id);
+
+-- Simple, fast policy
+CREATE POLICY "Precomputed access"
+ON documents FOR SELECT
+TO authenticated
+USING (
+  id IN (
+    SELECT document_id FROM user_document_access
+    WHERE user_id = auth.uid()
+  )
+);
+```
+
+---
+
+## SECURITY BEST PRACTICES
+
+### Always Enable RLS
+```sql
+-- Check tables without RLS
+SELECT tablename
+FROM pg_tables
+WHERE schemaname = 'public'
+AND tablename NOT IN (
+  SELECT tablename FROM pg_policies WHERE schemaname = 'public'
+);
+```
+
+### Default Deny
+```sql
+-- Enable RLS = default deny (no access without policy)
+ALTER TABLE sensitive_data ENABLE ROW LEVEL SECURITY;
+
+-- Only specific policies grant access
+CREATE POLICY "Explicit access only"
+ON sensitive_data FOR SELECT
+TO authenticated
+USING (/* specific conditions */);
+```
+
+### Avoid USING (true)
+```sql
+-- ❌ Dangerous: Opens access to all
+CREATE POLICY "Too permissive"
+ON users FOR ALL
+USING (true);
+
+-- ✅ Always specify conditions
+CREATE POLICY "Proper restriction"
+ON users FOR SELECT
+TO authenticated
+USING (id = auth.uid() OR is_public = true);
+```
+
+### Separate Policies by Operation
+```sql
+-- ✅ Granular control
+CREATE POLICY "Select policy" ON posts FOR SELECT ...;
+CREATE POLICY "Insert policy" ON posts FOR INSERT ...;
+CREATE POLICY "Update policy" ON posts FOR UPDATE ...;
+CREATE POLICY "Delete policy" ON posts FOR DELETE ...;
+
+-- ❌ Avoid overly broad policies
+CREATE POLICY "All operations" ON posts FOR ALL ...;
+```
+
+---
+
+## DEBUGGING RLS
+
+### Test Policies
+```sql
+-- Check what policies exist
+SELECT * FROM pg_policies WHERE tablename = 'posts';
+
+-- Test as specific user (Supabase)
+-- Use the SQL Editor with a specific user's JWT
+
+-- Debug query with RLS
+SET ROLE authenticated;
+SET request.jwt.claim.sub = 'user-uuid-here';
+SELECT * FROM posts;
+RESET ROLE;
+```
+
+### Common Issues
+1. **No data returned:** Check USING clause conditions
+2. **Can't insert:** Check WITH CHECK clause
+3. **Performance slow:** Add indexes on RLS filter columns
+4. **Bypass needed:** Use service_role (admin only)
+
+---
+
+## TESTING CHECKLIST
+
+- [ ] RLS enabled on all user-facing tables
+- [ ] Policies exist for all CRUD operations
+- [ ] Indexes created for policy filter columns
+- [ ] Tested with different user roles
+- [ ] Tested edge cases (no org, expired trial, etc.)
+- [ ] Performance tested with realistic data volume
+- [ ] service_role access restricted to backend only
+- [ ] No USING (true) on sensitive tables
+
+---
+
+**Reviewer:** ________ **Date:** ________
+**Security Audit:** [ ] PASS [ ] NEEDS REVIEW