dubs-server 1.0.0

package/documentation/KEEPER_RECOVERY_GUIDE.md

@@ -0,0 +1,381 @@
+# 🔄 Keeper Recovery Guide
+
+**Last Updated:** November 17, 2025  
+**System:** Dubs Jackpot Keeper with State Machine  
+**Status:** ✅ Production Ready with Auto-Recovery
+
+---
+
+## 🎯 TL;DR - Quick Recovery
+
+**Keeper stopped working?**
+
+```bash
+# 99% of the time, just restart it:
+heroku ps:restart jackpot-keeper -a dubs-server-dev
+
+# Watch it auto-recover:
+heroku logs --tail -a dubs-server-dev --dyno=jackpot-keeper
+```
+
+**That's it!** The state machine handles the rest automatically.
+
+---
+
+## 🔍 How Auto-Recovery Works
+
+### **On Every Keeper Startup:**
+
+```javascript
+🚀 Keeper bot started!
+
+🔧 Checking for stuck rounds...
+✅ No stuck rounds found
+
+// If stuck rounds exist:
+⚠️  Found 1 stuck round(s)!
+🔧 Recovering round 555 stuck in 'locked' for 127s
+   On-chain status: Locked, DB status: locked
+   Retrying reveal...
+✅ Randomness revealed!
+💰 Resolving round 555...
+🏆 Round 555 resolved!
+✅ Recovery complete
+```
+
+### **What It Checks:**
+
+1. **Database query:** Find rounds stuck in incomplete states
+2. **Time check:** Find rounds not updated in 5+ minutes  
+3. **On-chain verification:** Compare DB state vs blockchain
+4. **Smart resume:** Continue from exact step where it stopped
+
+---
+
+## 📋 Common Scenarios & Solutions
+
+### **Scenario 1: Keeper Crashed During Lock**
+
+**Symptoms:**
+- Round shows "Open" on-chain
+- Database shows "locking" status
+- No activity for 5+ minutes
+
+**Solution:**
+```bash
+# Just restart
+heroku ps:restart jackpot-keeper -a dubs-server-dev
+```
+
+**What Happens:**
+- Keeper detects round in "locking" state
+- Checks on-chain: still "Open"
+- Retries lock transaction
+- Continues to reveal → resolve → reset
+
+---
+
+### **Scenario 2: Keeper Crashed During Reveal**
+
+**Symptoms:**
+- Round shows "Locked" on-chain
+- Database shows "locked" or "revealing"
+- VRF not yet revealed
+
+**Solution:**
+```bash
+# Just restart
+heroku ps:restart jackpot-keeper -a dubs-server-dev
+```
+
+**What Happens:**
+- Keeper sees round is "locked"
+- Skips lock step (already done)
+- Calls reveal with new oracle seed
+- Continues to resolve → reset
+
+---
+
+### **Scenario 3: Keeper Crashed During Resolve**
+
+**Symptoms:**
+- Round shows "Locked" on-chain
+- VRF is revealed
+- Winner not yet paid
+
+**Solution:**
+```bash
+# Just restart
+heroku ps:restart jackpot-keeper -a dubs-server-dev
+```
+
+**What Happens:**
+- Keeper sees round is "locked" with VRF
+- Skips lock and reveal (already done)
+- Calls resolve to pay winner
+- Resets for next round
+
+---
+
+### **Scenario 4: Keeper is Running but Not Processing**
+
+**Check if it's actually stuck:**
+```bash
+# 1. Check for stuck rounds
+curl https://dubs-server-dev-55d1fba09a97.herokuapp.com/api/keeper/rounds/stuck
+
+# 2. Check keeper health
+curl https://dubs-server-dev-55d1fba09a97.herokuapp.com/api/keeper/health
+
+# 3. Check logs
+heroku logs --tail -a dubs-server-dev --dyno=jackpot-keeper
+```
+
+**If stuck rounds found:**
+```bash
+# Restart to trigger recovery
+heroku ps:restart jackpot-keeper -a dubs-server-dev
+```
+
+---
+
+### **Scenario 5: Database Shows Stuck Round, On-Chain Says Open**
+
+**This means round completed but DB didn't update**
+
+**Solution:**
+```bash
+# Manually mark as resolved
+heroku pg:psql -a dubs-server-dev -c "
+UPDATE keeper_rounds 
+SET status = 'resolved', resolved_at = NOW()
+WHERE round_id = 555;
+"
+
+# Restart keeper
+heroku ps:restart jackpot-keeper -a dubs-server-dev
+```
+
+---
+
+## 🚨 Emergency Recovery (Nuclear Option)
+
+**If keeper is completely broken and won't recover:**
+
+### **Step 1: Stop Keeper**
+```bash
+heroku ps:scale jackpot-keeper=0 -a dubs-server-dev
+```
+
+### **Step 2: Check On-Chain State**
+```bash
+curl https://dubs-server-dev-55d1fba09a97.herokuapp.com/jackpot/round/current
+```
+
+### **Step 3: Clean Database**
+```bash
+# Mark all incomplete rounds as resolved
+heroku pg:psql -a dubs-server-dev -c "
+UPDATE keeper_rounds 
+SET status = 'resolved', resolved_at = NOW()
+WHERE status NOT IN ('resolved', 'open');
+"
+
+# Check what's in DB now
+heroku pg:psql -a dubs-server-dev -c "
+SELECT round_id, status FROM keeper_rounds 
+WHERE status != 'resolved' 
+ORDER BY round_id DESC;
+"
+```
+
+### **Step 4: Restart Keeper**
+```bash
+heroku ps:scale jackpot-keeper=1 -a dubs-server-dev
+heroku logs --tail -a dubs-server-dev --dyno=jackpot-keeper
+```
+
+---
+
+## 📊 Monitoring Commands
+
+### **Check if Keeper is Alive:**
+```bash
+# Method 1: Check Heroku process
+heroku ps -a dubs-server-dev | grep jackpot-keeper
+
+# Method 2: Check API
+curl https://dubs-server-dev-55d1fba09a97.herokuapp.com/api/keeper/health
+
+# Method 3: Check logs (should see activity every 5 seconds)
+heroku logs --tail -a dubs-server-dev --dyno=jackpot-keeper -n 10
+```
+
+### **Check for Problems:**
+```bash
+# Stuck rounds?
+curl https://dubs-server-dev-55d1fba09a97.herokuapp.com/api/keeper/rounds/stuck
+
+# Recent failures?
+heroku pg:psql -a dubs-server-dev -c "
+SELECT action, success, error_message, timestamp
+FROM keeper_actions
+WHERE success = false
+ORDER BY timestamp DESC
+LIMIT 10;
+"
+
+# High retry counts?
+heroku pg:psql -a dubs-server-dev -c "
+SELECT round_id, status, retry_count, last_error
+FROM keeper_rounds
+WHERE retry_count > 0
+ORDER BY round_id DESC
+LIMIT 10;
+"
+```
+
+---
+
+## 🔧 Troubleshooting Checklist
+
+**Keeper won't start:**
+- [ ] Check Heroku dyno is running: `heroku ps -a dubs-server-dev`
+- [ ] Check DATABASE_URL is set: `heroku config:get DATABASE_URL -a dubs-server-dev`
+- [ ] Check KEEPER_PRIVATE_KEY is set: `heroku config:get KEEPER_PRIVATE_KEY -a dubs-server-dev`
+- [ ] Check logs for errors: `heroku logs --tail -a dubs-server-dev --dyno=jackpot-keeper`
+
+**Keeper running but not processing:**
+- [ ] Check current round: `curl https://dubs-server-dev-55d1fba09a97.herokuapp.com/jackpot/round/current`
+- [ ] Check stuck rounds: `curl https://dubs-server-dev-55d1fba09a97.herokuapp.com/api/keeper/rounds/stuck`
+- [ ] Restart keeper: `heroku ps:restart jackpot-keeper -a dubs-server-dev`
+
+**Round stuck for hours:**
+- [ ] Check database state vs on-chain state
+- [ ] Manually mark as resolved (see Scenario 5 above)
+- [ ] Restart keeper
+
+---
+
+## 💊 Health Check Script
+
+**Save this as `check-keeper.sh`:**
+
+```bash
+#!/bin/bash
+
+echo "🔍 KEEPER HEALTH CHECK"
+echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+
+# Check if keeper dyno is running
+echo "📊 Dyno Status:"
+heroku ps -a dubs-server-dev | grep jackpot-keeper
+
+# Check API health
+echo -e "\n💊 API Health:"
+HEALTH=$(curl -s https://dubs-server-dev-55d1fba09a97.herokuapp.com/api/keeper/health)
+echo $HEALTH | jq '{healthy, stuckRounds: .stuckRounds | length, resolved: .summary.resolved_rounds}'
+
+# Check for stuck rounds
+STUCK=$(echo $HEALTH | jq -r '.stuckRounds | length')
+if [ "$STUCK" != "0" ]; then
+  echo "⚠️  WARNING: $STUCK stuck round(s) detected!"
+  echo "   Action: heroku ps:restart jackpot-keeper -a dubs-server-dev"
+else
+  echo "✅ No stuck rounds"
+fi
+
+# Check recent activity
+echo -e "\n📝 Recent Activity:"
+heroku logs -a dubs-server-dev --dyno=jackpot-keeper -n 3 | grep -E "(Round|✅|❌)"
+
+echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+```
+
+**Run it:**
+```bash
+chmod +x check-keeper.sh
+./check-keeper.sh
+```
+
+---
+
+## 🎯 Performance Metrics
+
+**Check how well the keeper is performing:**
+
+```bash
+heroku pg:psql -a dubs-server-dev -c "
+SELECT 
+  action,
+  COUNT(*) as total,
+  SUM(CASE WHEN success THEN 1 ELSE 0 END) as successes,
+  ROUND(100.0 * SUM(CASE WHEN success THEN 1 ELSE 0 END) / COUNT(*), 1) as success_rate,
+  ROUND(AVG(duration_ms)) as avg_ms,
+  MAX(duration_ms) as max_ms
+FROM keeper_actions
+GROUP BY action
+ORDER BY action;
+"
+```
+
+**Expected output:**
+```
+ action  | total | successes | success_rate | avg_ms | max_ms 
+---------|-------|-----------|--------------|--------|--------
+ lock    |    10 |        10 |        100.0 |   1352 |   2000
+ reset   |    40 |        40 |        100.0 |   1255 |   2100
+ resolve |    10 |        10 |        100.0 |   1740 |   2500
+ reveal  |    10 |        10 |        100.0 |    966 |   1500
+```
+
+**Good:** 95%+ success rate, <3s average  
+**Concerning:** <90% success rate, >5s average → investigate
+
+---
+
+## 📞 When to Ask for Help
+
+**Auto-recovery should handle:**
+- ✅ Keeper crashes
+- ✅ Network timeouts
+- ✅ Single transaction failures
+- ✅ Heroku dyno restarts
+
+**You need manual intervention if:**
+- ❌ Same round stuck for 30+ minutes after restart
+- ❌ Success rate drops below 80%
+- ❌ Database shows many retries (retry_count > 5)
+- ❌ On-chain state and DB state are completely mismatched
+
+---
+
+## 🎉 The Power of State Machine
+
+**Before State Machine:**
+- Keeper crash = manual recovery needed
+- No visibility into what failed
+- Lost track of progress
+- Had to debug from logs
+
+**With State Machine:**
+- Keeper crash = automatic recovery
+- Complete visibility in database  
+- Never loses progress
+- Easy debugging with SQL queries
+
+**Recovery went from "panic and debug for 2 hours" to "restart and it fixes itself in 10 seconds"**
+
+---
+
+## 📚 Additional Resources
+
+- Database Queries: See `DATABASE_QUERIES.md`
+- State Machine Guide: See `KEEPER_STATE_MACHINE.md`
+- Implementation Details: See `STATE_MACHINE_IMPLEMENTATION.md`
+- Monitoring API: `https://dubs-server-dev-55d1fba09a97.herokuapp.com/api/keeper/*`
+
+---
+
+**Your keeper is bulletproof. It will auto-recover from 99% of failures.** 🛡️
+

package/documentation/KEEPER_SETUP.md

@@ -0,0 +1,206 @@
+# 🤖 Keeper Bot Setup for Heroku
+
+## The Problem
+
+The keeper bot needs a wallet to sign transactions, but wallet files in `wallets/` directory are ignored by git and won't be deployed to Heroku.
+
+## The Solution
+
+We use an environment variable `KEEPER_PRIVATE_KEY` to securely pass the wallet's private key to Heroku.
+
+---
+
+## Quick Setup (Automated)
+
+```bash
+cd /Users/adamdahan/Developer/iheartsolana/solana-programs/dubs-server
+
+# Run the setup script
+./setup-heroku-keeper.sh
+
+# That's it! The script will:
+# 1. Read wallets/jackpot_oracle.json
+# 2. Set KEEPER_PRIVATE_KEY on Heroku
+# 3. Show you the wallet address
+```
+
+---
+
+## Manual Setup
+
+If you prefer to do it manually:
+
+### Step 1: Get the Private Key
+
+```bash
+cd /Users/adamdahan/Developer/iheartsolana/solana-programs/dubs-server
+
+# View the wallet (it's a JSON array)
+cat wallets/jackpot_oracle.json
+```
+
+You'll see something like:
+```json
+[135,155,140,135,170,89,66,124,...]
+```
+
+### Step 2: Set on Heroku
+
+```bash
+# Set the environment variable (paste the entire JSON array)
+heroku config:set KEEPER_PRIVATE_KEY='[135,155,140,135,170,89,66,124,...]' -a dubs-server-dev
+```
+
+**Important:** Keep the single quotes around the JSON array!
+
+### Step 3: Verify
+
+```bash
+# Check it's set (will show first few characters)
+heroku config:get KEEPER_PRIVATE_KEY -a dubs-server-dev
+
+# Deploy and test
+git push heroku-dev main
+heroku ps:scale jackpot-keeper=1 -a dubs-server-dev
+heroku logs --tail --ps jackpot-keeper -a dubs-server-dev
+```
+
+---
+
+## How It Works
+
+The keeper bot now checks for the wallet in this order:
+
+1. **Environment variable** (`KEEPER_PRIVATE_KEY`) - for Heroku
+2. **Oracle wallet file** (`wallets/jackpot_oracle.json`) - for local development  
+3. **Main wallet file** (`~/.config/solana/id.json`) - fallback for local
+
+This means:
+- ✅ Works on Heroku (uses env var)
+- ✅ Works locally (uses wallet file)
+- ✅ Secure (private key not in git)
+
+---
+
+## Verification
+
+After deployment, check the logs:
+
+```bash
+heroku logs --tail --ps jackpot-keeper -a dubs-server-dev
+```
+
+You should see:
+```
+📂 Using wallet from KEEPER_PRIVATE_KEY environment variable
+🤖 Complete Jackpot Keeper Bot
+   Wallet: FWUJCthDfPcgmTvdQWM5uofxxiYjqJFMMwiLYvS7LBFa
+
+🚀 Keeper bot started!
+```
+
+---
+
+## Troubleshooting
+
+### Error: "No wallet found"
+
+```bash
+# Make sure env var is set
+heroku config:get KEEPER_PRIVATE_KEY -a dubs-server-dev
+
+# If empty, run setup again
+./setup-heroku-keeper.sh
+```
+
+### Error: "Invalid private key"
+
+The JSON array must be exactly as it appears in the wallet file:
+- Start with `[`
+- End with `]`
+- 64 numbers separated by commas
+- No extra spaces or newlines
+
+### Keeper crashes on startup
+
+```bash
+# Check the logs for the exact error
+heroku logs --tail --ps jackpot-keeper -a dubs-server-dev
+
+# Restart it
+heroku ps:restart jackpot-keeper -a dubs-server-dev
+```
+
+---
+
+## Security Notes
+
+⚠️ **Important Security Considerations:**
+
+1. **Never commit wallet files to git** (they're already in `.gitignore`)
+2. **Never share the private key** in public channels
+3. **Use different wallets for different environments** (devnet vs mainnet)
+4. **Rotate keys regularly** on mainnet
+5. **Monitor wallet balance** - keeper needs SOL for transaction fees
+
+---
+
+## Wallet Info
+
+The keeper uses the oracle wallet:
+
+- **Address:** `FWUJCthDfPcgmTvdQWM5uofxxiYjqJFMMwiLYvS7LBFa`
+- **File:** `wallets/jackpot_oracle.json` (local only)
+- **Purpose:** Signs keeper transactions (open, lock, resolve rounds)
+- **Network:** Devnet (test SOL only)
+
+---
+
+## For Mainnet (Future)
+
+When deploying to mainnet:
+
+1. **Create a new mainnet wallet:**
+   ```bash
+   solana-keygen new --outfile wallets/jackpot_oracle_mainnet.json
+   ```
+
+2. **Fund it with real SOL:**
+   ```bash
+   solana transfer <NEW_WALLET_ADDRESS> 1.0 --url mainnet-beta
+   ```
+
+3. **Set on production Heroku:**
+   ```bash
+   PRIVATE_KEY=$(cat wallets/jackpot_oracle_mainnet.json)
+   heroku config:set KEEPER_PRIVATE_KEY="$PRIVATE_KEY" -a dubs-server-prod
+   ```
+
+4. **Monitor balance regularly** - keeper pays gas fees
+
+---
+
+## Quick Reference
+
+```bash
+# Setup keeper wallet
+./setup-heroku-keeper.sh
+
+# Deploy keeper
+git push heroku-dev main
+heroku ps:scale jackpot-keeper=1 -a dubs-server-dev
+
+# Check logs
+heroku logs --tail --ps jackpot-keeper -a dubs-server-dev
+
+# Check wallet balance
+solana balance FWUJCthDfPcgmTvdQWM5uofxxiYjqJFMMwiLYvS7LBFa --url devnet
+
+# Restart keeper
+heroku ps:restart jackpot-keeper -a dubs-server-dev
+```
+
+---
+
+**Built with ❤️ on Solana** 🚀🎰
+