dubs-server 1.0.0

package/documentation/DATABASE_QUERIES.md

@@ -0,0 +1,269 @@
+# 📊 Heroku Database Quick Reference
+
+## 🔗 Connect to Database
+
+### Interactive psql Shell:
+```bash
+heroku pg:psql -a dubs-server-dev
+```
+
+### Run Single Query:
+```bash
+heroku pg:psql -a dubs-server-dev -c "YOUR QUERY HERE"
+```
+
+### Get Database Info:
+```bash
+heroku pg:info -a dubs-server-dev
+```
+
+---
+
+## 🎯 Useful Queries
+
+### 1. View Recent Rounds
+```sql
+SELECT round_id, status, entry_count, retry_count, 
+       locked_at, revealed_at, resolved_at
+FROM keeper_rounds 
+ORDER BY round_id DESC 
+LIMIT 10;
+```
+
+### 2. View Rounds with Full Verification Data
+```sql
+SELECT r.round_id, r.winner, r.win_amount, r.total_pot, 
+       r.server_seed_hash, r.oracle_seed, r.signature
+FROM jackpot_rounds r
+WHERE r.server_seed_hash IS NOT NULL 
+  AND r.oracle_seed IS NOT NULL
+ORDER BY r.timestamp DESC
+LIMIT 10;
+```
+
+### 3. Check Action Log for a Round
+```sql
+SELECT action, success, duration_ms, timestamp, error_message
+FROM keeper_actions 
+WHERE round_id = 551
+ORDER BY timestamp ASC;
+```
+
+### 4. See Stuck Rounds (View)
+```sql
+SELECT * FROM stuck_rounds;
+```
+
+### 5. Overall Health Summary (View)
+```sql
+SELECT * FROM keeper_health_summary;
+```
+
+### 6. Recent Keeper Health Snapshots
+```sql
+SELECT rounds_completed, consecutive_failures, 
+       last_success_round, uptime_seconds, timestamp
+FROM keeper_health 
+ORDER BY timestamp DESC 
+LIMIT 10;
+```
+
+### 7. Success Rate Analysis
+```sql
+SELECT 
+  action,
+  COUNT(*) as total_attempts,
+  SUM(CASE WHEN success THEN 1 ELSE 0 END) as successes,
+  ROUND(100.0 * SUM(CASE WHEN success THEN 1 ELSE 0 END) / COUNT(*), 2) as success_rate_pct,
+  AVG(duration_ms) as avg_duration_ms
+FROM keeper_actions
+GROUP BY action
+ORDER BY action;
+```
+
+### 8. Find Slow Operations
+```sql
+SELECT round_id, action, duration_ms, timestamp
+FROM keeper_actions
+WHERE duration_ms > 2000  -- Slower than 2 seconds
+ORDER BY duration_ms DESC
+LIMIT 10;
+```
+
+### 9. Find Rounds with Errors
+```sql
+SELECT DISTINCT round_id, status, retry_count, last_error
+FROM keeper_rounds
+WHERE retry_count > 0
+ORDER BY round_id DESC
+LIMIT 10;
+```
+
+### 10. Rounds Completed in Last Hour
+```sql
+SELECT round_id, 
+       EXTRACT(EPOCH FROM (resolved_at - locked_at)) as completion_seconds
+FROM keeper_rounds
+WHERE resolved_at > NOW() - INTERVAL '1 hour'
+ORDER BY resolved_at DESC;
+```
+
+---
+
+## 🔍 Debugging Queries
+
+### Find What Round is Currently Processing:
+```sql
+SELECT round_id, status, 
+       EXTRACT(EPOCH FROM (NOW() - updated_at)) as seconds_since_update
+FROM keeper_rounds
+WHERE status NOT IN ('resolved', 'open')
+ORDER BY updated_at DESC
+LIMIT 5;
+```
+
+### See Complete Round Lifecycle:
+```sql
+-- Detailed timeline for round 551
+SELECT 
+  'Created' as phase, created_at as timestamp FROM keeper_rounds WHERE round_id = 551
+UNION ALL
+SELECT 'Locked', locked_at FROM keeper_rounds WHERE round_id = 551
+UNION ALL
+SELECT 'Revealed', revealed_at FROM keeper_rounds WHERE round_id = 551
+UNION ALL  
+SELECT 'Resolved', resolved_at FROM keeper_rounds WHERE round_id = 551
+UNION ALL
+SELECT 'Reset', reset_at FROM keeper_rounds WHERE round_id = 551
+ORDER BY timestamp;
+```
+
+### Find Longest Round Completion:
+```sql
+SELECT round_id,
+       EXTRACT(EPOCH FROM (resolved_at - locked_at)) as seconds_to_complete
+FROM keeper_rounds
+WHERE resolved_at IS NOT NULL
+ORDER BY seconds_to_complete DESC
+LIMIT 10;
+```
+
+---
+
+## 📊 Monitoring Queries (Run Periodically)
+
+### Check System Health:
+```bash
+heroku pg:psql -a dubs-server-dev -c "
+SELECT 
+  (SELECT COUNT(*) FROM stuck_rounds) as stuck_count,
+  (SELECT COUNT(*) FROM keeper_rounds WHERE status = 'resolved') as resolved_count,
+  (SELECT AVG(EXTRACT(EPOCH FROM (resolved_at - locked_at))) 
+   FROM keeper_rounds 
+   WHERE resolved_at IS NOT NULL) as avg_completion_seconds;
+"
+```
+
+### Recent Activity:
+```bash
+heroku pg:psql -a dubs-server-dev -c "
+SELECT action, success, COUNT(*) as count, AVG(duration_ms) as avg_ms
+FROM keeper_actions
+WHERE timestamp > NOW() - INTERVAL '1 hour'
+GROUP BY action, success
+ORDER BY action, success DESC;
+"
+```
+
+---
+
+## 🛠️ Maintenance Queries
+
+### Clear Old Action Logs (Keep last 1000):
+```sql
+DELETE FROM keeper_actions
+WHERE id NOT IN (
+  SELECT id FROM keeper_actions 
+  ORDER BY timestamp DESC 
+  LIMIT 1000
+);
+```
+
+### Mark Stuck Round as Resolved (Manual Override):
+```sql
+UPDATE keeper_rounds
+SET status = 'resolved', resolved_at = NOW()
+WHERE round_id = 999;
+```
+
+### Reset Retry Counter:
+```sql
+UPDATE keeper_rounds
+SET retry_count = 0, last_error = NULL
+WHERE round_id = 999;
+```
+
+---
+
+## 🚀 Quick Commands You'll Use Often
+
+**See what keeper is doing right now:**
+```bash
+heroku pg:psql -a dubs-server-dev -c "SELECT round_id, status, EXTRACT(EPOCH FROM (NOW() - updated_at)) as secs_ago FROM keeper_rounds ORDER BY updated_at DESC LIMIT 5;"
+```
+
+**Check if any rounds are stuck:**
+```bash
+heroku pg:psql -a dubs-server-dev -c "SELECT * FROM stuck_rounds;"
+```
+
+**See recent completions:**
+```bash
+heroku pg:psql -a dubs-server-dev -c "SELECT round_id, status, entry_count FROM keeper_rounds WHERE status = 'resolved' ORDER BY resolved_at DESC LIMIT 5;"
+```
+
+**Full dashboard view:**
+```bash
+curl -s https://dubs-server-dev-55d1fba09a97.herokuapp.com/api/keeper/dashboard | jq .
+```
+
+---
+
+## 💡 Pro Tips
+
+1. **Watch Live Updates:**
+```bash
+watch -n 2 'heroku pg:psql -a dubs-server-dev -c "SELECT round_id, status FROM keeper_rounds ORDER BY round_id DESC LIMIT 5"'
+```
+
+2. **Export to CSV:**
+```bash
+heroku pg:psql -a dubs-server-dev -c "COPY (SELECT * FROM keeper_rounds ORDER BY round_id DESC LIMIT 100) TO STDOUT WITH CSV HEADER" > rounds.csv
+```
+
+3. **Backup Important Data:**
+```bash
+heroku pg:backups:capture -a dubs-server-dev
+heroku pg:backups:download -a dubs-server-dev
+```
+
+---
+
+## 📱 Access from Any Tool
+
+**Get DATABASE_URL:**
+```bash
+heroku config:get DATABASE_URL -a dubs-server-dev
+```
+
+Then use with any Postgres client:
+- TablePlus
+- pgAdmin
+- DBeaver
+- psql locally
+
+**Quick connect string format:**
+```
+postgresql://username:password@host:5432/database?sslmode=require
+```
+

package/documentation/DATABASE_RETENTION_POLICY.md

@@ -0,0 +1,390 @@
+# 📦 Database Retention & Cleanup Policy
+
+**Last Updated:** November 18, 2025  
+**Purpose:** Prevent unbounded database growth while preserving audit trail
+
+---
+
+## 🎯 Retention Policies
+
+### **Forever (Never Delete):**
+
+1. **`jackpot_rounds`** - Historical winners
+   - **Why:** Users need to verify ANY past round
+   - **Size:** ~500 bytes per round
+   - **Growth:** ~50 rounds/day = 25 KB/day = 9 MB/year
+   - **Acceptable:** Plenty of room on Heroku Postgres
+
+2. **`keeper_rounds` (resolved status)** - Completed rounds
+   - **Why:** Audit trail for completed rounds
+   - **Size:** ~300 bytes per round
+   - **Growth:** ~50 rounds/day = 15 KB/day = 5.5 MB/year
+   - **Acceptable:** Essential audit data
+
+### **30 Days Retention:**
+
+3. **`keeper_actions`** - Detailed action logs
+   - **Why:** Recent debugging, older logs less useful
+   - **Size:** ~200 bytes per action
+   - **Growth:** ~200 actions/day = 40 KB/day
+   - **Cleanup:** Delete actions older than 30 days
+   - **Impact:** Keeps last ~6,000 actions (~1.2 MB)
+
+4. **`keeper_rounds` (incomplete)** - Stuck/failed rounds
+   - **Why:** Old stuck rounds aren't relevant
+   - **Size:** ~300 bytes per round
+   - **Growth:** Should be 0 (no stuck rounds!)
+   - **Cleanup:** Delete incomplete rounds older than 30 days
+
+### **90 Days Retention:**
+
+5. **`keeper_health`** - Health snapshots
+   - **Why:** Long-term trend analysis
+   - **Size:** ~150 bytes per snapshot
+   - **Growth:** ~1,440 snapshots/day (every minute) = 216 KB/day
+   - **Cleanup:** Delete snapshots older than 90 days
+   - **Impact:** Keeps ~130,000 snapshots (~19.5 MB)
+
+---
+
+## 📊 Growth Projections
+
+### **Without Cleanup:**
+
+| Table | Per Day | Per Month | Per Year |
+|-------|---------|-----------|----------|
+| jackpot_rounds | 25 KB | 750 KB | 9 MB |
+| keeper_rounds | 15 KB | 450 KB | 5.5 MB |
+| keeper_actions | 40 KB | 1.2 MB | **14.6 MB** |
+| keeper_health | 216 KB | 6.5 MB | **78.8 MB** |
+| **TOTAL** | **296 KB** | **8.9 MB** | **108 MB** |
+
+**Conclusion:** Without cleanup, ~108 MB/year. Heroku has 1 GB limit. Would take **9+ years** to fill up!
+
+### **With Cleanup:**
+
+| Table | Max Size (Steady State) |
+|-------|-------------------------|
+| jackpot_rounds | ~9 MB/year (grows forever) |
+| keeper_rounds | ~5.5 MB/year (grows forever) |
+| keeper_actions | ~1.2 MB (capped at 30 days) |
+| keeper_health | ~19.5 MB (capped at 90 days) |
+| **TOTAL** | **~35 MB steady state** |
+
+**Conclusion:** With cleanup, stays under 50 MB indefinitely. **No storage concerns for years.**
+
+---
+
+## 🔄 Automated Cleanup (Heroku Scheduler)
+
+### **Setup Once:**
+
+```bash
+# Add Heroku Scheduler addon (free!)
+heroku addons:create scheduler:standard -a dubs-server-dev
+
+# Open scheduler dashboard
+heroku addons:open scheduler -a dubs-server-dev
+```
+
+### **Add Daily Job:**
+
+**In Heroku Scheduler Dashboard:**
+- **Command:** `node scripts/cleanup-database.js`
+- **Frequency:** Daily
+- **Time:** 3:00 AM UTC (off-peak)
+- **Dyno Size:** Standard-1X
+
+**Or via CLI:**
+```bash
+heroku addons:create scheduler:standard -a dubs-server-dev
+# Then configure in dashboard
+```
+
+---
+
+## 🛠️ Manual Cleanup
+
+### **Run Cleanup Now:**
+
+```bash
+# On Heroku
+heroku run node scripts/cleanup-database.js -a dubs-server-dev
+
+# Locally (if DATABASE_URL set)
+node scripts/cleanup-database.js
+```
+
+### **Check Space Usage:**
+
+```bash
+heroku pg:info -a dubs-server-dev
+```
+
+Look for:
+```
+Data Size: 7.93 MB / 1 GB (0.77%)
+```
+
+**When to worry:** > 500 MB (50%) → Increase cleanup frequency
+
+---
+
+## 🔍 What Gets Cleaned
+
+### **keeper_actions** (30 days)
+
+**Keeps:**
+- Last 30 days of action logs
+- ~6,000 recent actions
+- All recent debugging data
+
+**Deletes:**
+- Actions older than 30 days
+- Old success logs (not needed)
+- Old error logs (already investigated)
+
+**Example:** Round 100 completed 35 days ago → action logs deleted, but round data kept
+
+### **keeper_health** (90 days)
+
+**Keeps:**
+- Last 90 days of health snapshots
+- ~130,000 data points
+- Enough for trend analysis
+
+**Deletes:**
+- Snapshots older than 90 days
+- Ancient performance data
+
+### **keeper_rounds incomplete** (30 days)
+
+**Keeps:**
+- All resolved rounds (forever)
+- Recent stuck rounds (for debugging)
+
+**Deletes:**
+- Stuck rounds older than 30 days
+- Failed rounds that never completed
+
+**Why:** If round stuck for 30 days, it's never completing!
+
+---
+
+## 📈 Monitoring Cleanup
+
+### **Check if Cleanup is Running:**
+
+```bash
+# View scheduler logs
+heroku logs --ps scheduler -a dubs-server-dev
+
+# Check last cleanup time
+heroku pg:psql -a dubs-server-dev -c "
+SELECT MAX(timestamp) as last_cleanup 
+FROM keeper_actions 
+WHERE action = 'cleanup';
+"
+```
+
+### **See What Would Be Deleted (Dry Run):**
+
+```bash
+heroku pg:psql -a dubs-server-dev -c "
+SELECT 
+  'keeper_actions' as table_name,
+  COUNT(*) as rows_to_delete
+FROM keeper_actions
+WHERE timestamp < NOW() - INTERVAL '30 days'
+
+UNION ALL
+
+SELECT 
+  'keeper_health',
+  COUNT(*)
+FROM keeper_health
+WHERE timestamp < NOW() - INTERVAL '90 days'
+
+UNION ALL
+
+SELECT 
+  'keeper_rounds (incomplete)',
+  COUNT(*)
+FROM keeper_rounds
+WHERE status NOT IN ('resolved', 'open')
+  AND updated_at < NOW() - INTERVAL '30 days';
+"
+```
+
+---
+
+## 🚨 Emergency: Database Full
+
+**If you somehow reach 90%+ capacity:**
+
+### **Option 1: Aggressive Cleanup**
+
+```bash
+# Delete actions older than 7 days instead of 30
+heroku pg:psql -a dubs-server-dev -c "
+DELETE FROM keeper_actions
+WHERE timestamp < NOW() - INTERVAL '7 days';
+"
+
+# Delete health older than 30 days instead of 90
+heroku pg:psql -a dubs-server-dev -c "
+DELETE FROM keeper_health
+WHERE timestamp < NOW() - INTERVAL '30 days';
+"
+
+# Vacuum to reclaim space
+heroku pg:psql -a dubs-server-dev -c "VACUUM FULL;"
+```
+
+### **Option 2: Export & Archive**
+
+```bash
+# Export old data to CSV
+heroku pg:psql -a dubs-server-dev -c "
+COPY (
+  SELECT * FROM keeper_actions 
+  WHERE timestamp < NOW() - INTERVAL '90 days'
+) TO STDOUT WITH CSV HEADER
+" > keeper_actions_archive_$(date +%Y%m%d).csv
+
+# Store in S3, Google Cloud Storage, or local
+
+# Then delete from database
+heroku pg:psql -a dubs-server-dev -c "
+DELETE FROM keeper_actions
+WHERE timestamp < NOW() - INTERVAL '90 days';
+"
+```
+
+### **Option 3: Upgrade Database**
+
+```bash
+# Upgrade to larger plan
+heroku addons:upgrade postgresql-colorful-22525:standard-0 -a dubs-server-dev
+
+# Plans:
+# essential-0: 1 GB (current, $5/month)
+# essential-1: 10 GB ($50/month)
+# standard-0: 64 GB ($200/month)
+```
+
+---
+
+## 🎯 Recommended Strategy
+
+### **For Devnet/Testing:**
+
+Run cleanup **weekly**:
+```bash
+# Add to Heroku Scheduler
+node scripts/cleanup-database.js
+# Frequency: Daily
+```
+
+**Why:** Development generates lots of test data, clean it aggressively
+
+### **For Production/Mainnet:**
+
+Run cleanup **daily** with current policies:
+- keeper_actions: 30 days
+- keeper_health: 90 days  
+- keeper_rounds: Keep resolved forever
+
+**Why:** Balance between audit trail and storage
+
+### **For High Volume (1000+ rounds/day):**
+
+**Option A:** More aggressive retention
+- keeper_actions: 14 days
+- keeper_health: 30 days
+- Consider archiving to S3
+
+**Option B:** Upgrade database
+- Move to 10 GB plan ($50/month)
+- Keep longer retention
+
+---
+
+## 📊 Cost Analysis
+
+### **Storage Costs:**
+
+**Heroku Postgres Essential-0 (1 GB):**
+- Cost: $5/month
+- Capacity: 1 GB
+- Expected usage with cleanup: <50 MB
+- **Verdict:** Plenty of room ✅
+
+**Upgrade triggers:**
+- > 500 MB used → Consider cleanup tuning
+- > 800 MB used → Upgrade to Essential-1 (10 GB, $50/mo)
+
+### **Cleanup Costs:**
+
+**Heroku Scheduler:**
+- Cost: FREE (included)
+- Runs: Daily at 3am
+- Duration: ~30 seconds
+- Dyno hours: ~0.5 hours/month
+- **Verdict:** Free tier covers it ✅
+
+---
+
+## 🧪 Test Cleanup Locally
+
+```bash
+# Set DATABASE_URL to local Postgres
+export DATABASE_URL="postgresql://localhost/dubs_test"
+
+# Run cleanup
+node scripts/cleanup-database.js
+
+# Expected output:
+# 🧹 DATABASE CLEANUP STARTING
+# 📊 Before Cleanup: ...
+# 🧹 Cleaning keeper_actions...
+#    Deleted 0 old action logs
+# ✅ CLEANUP COMPLETE
+```
+
+---
+
+## 📋 Cleanup Checklist
+
+**Initial Setup (One Time):**
+- [ ] Deploy cleanup script: `git push heroku-dev main`
+- [ ] Add Heroku Scheduler addon
+- [ ] Configure daily job at 3am
+- [ ] Test run: `heroku run node scripts/cleanup-database.js -a dubs-server-dev`
+
+**Monthly Monitoring:**
+- [ ] Check database size: `heroku pg:info -a dubs-server-dev`
+- [ ] Verify cleanup runs: `heroku logs --ps scheduler -a dubs-server-dev`
+- [ ] Review retention policies (adjust if needed)
+
+**Yearly Review:**
+- [ ] Analyze growth trends
+- [ ] Adjust retention policies
+- [ ] Consider database upgrade
+- [ ] Archive old data if needed
+
+---
+
+## 🎯 Current Recommendation
+
+**For your devnet testing:**
+
+✅ **Run cleanup weekly** (daily is overkill for low volume)  
+✅ **Keep current retention policies** (30/90 days)  
+✅ **Monitor size monthly**  
+✅ **Don't worry about storage** (you have 1000x headroom)
+
+**The database will NOT fill up.** Even at 1000 rounds/day, you have years before hitting limits.
+
+But having automated cleanup is good practice and prevents surprises! 🎯
+