@dotdo/postgres 0.1.0

package/src/iceberg/SCHEDULER.md

@@ -0,0 +1,580 @@
+# Iceberg Snapshot Scheduler
+
+Automatic snapshot scheduling and event-triggered snapshots for Apache Iceberg tables.
+
+## Overview
+
+The Snapshot Scheduler provides:
+
+1. **Cron-based Scheduling** - Create snapshots on a schedule (hourly, daily, weekly, etc.)
+2. **Event-triggered Snapshots** - Create snapshots after a certain number of writes
+3. **Retention Policy Enforcement** - Automatically expire old snapshots
+4. **Cloudflare Integration** - Works with Cron Triggers and Durable Object alarms
+
+## Installation
+
+The scheduler is part of the `@dotdo/postgres` package:
+
+```typescript
+import {
+  SnapshotScheduler,
+  createSnapshotScheduler,
+} from '@dotdo/postgres/iceberg'
+```
+
+## Quick Start
+
+### Basic Usage
+
+```typescript
+import { createSnapshotScheduler } from '@dotdo/postgres/iceberg'
+
+// Create scheduler
+const scheduler = createSnapshotScheduler({
+  bucket: env.ICEBERG_BUCKET,
+  tablePath: 'iceberg',
+})
+
+await scheduler.initialize()
+
+// Schedule hourly snapshots
+await scheduler.schedule('events', '0 * * * *')
+
+// Set write threshold
+await scheduler.setThreshold('events', 10000)
+
+// Enforce retention policy
+await scheduler.enforceRetention('events', {
+  maxAgeDays: 30,
+  minSnapshotsToRetain: 5,
+})
+```
+
+### With Durable Objects
+
+```typescript
+import { DurableObject } from 'cloudflare:workers'
+import { createSnapshotScheduler } from '@dotdo/postgres/iceberg'
+
+export class SchedulerDO extends DurableObject {
+  async alarm() {
+    const scheduler = createSnapshotScheduler({
+      bucket: this.env.ICEBERG_BUCKET,
+      tablePath: 'iceberg',
+      state: this.ctx.state,
+    })
+
+    await scheduler.initialize()
+    await scheduler.processDueSnapshots()
+  }
+}
+```
+
+## Cron Expressions
+
+The scheduler uses standard cron format:
+
+```
+┌───────────── minute (0-59)
+│ ┌───────────── hour (0-23)
+│ │ ┌───────────── day of month (1-31)
+│ │ │ ┌───────────── month (1-12)
+│ │ │ │ ┌───────────── day of week (0-6, 0 = Sunday)
+│ │ │ │ │
+* * * * *
+```
+
+### Common Examples
+
+```typescript
+// Every hour at minute 0
+await scheduler.schedule('events', '0 * * * *')
+
+// Daily at midnight
+await scheduler.schedule('events', '0 0 * * *')
+
+// Weekly on Sunday at midnight
+await scheduler.schedule('events', '0 0 * * 0')
+
+// Monthly on the 1st at midnight
+await scheduler.schedule('events', '0 0 1 * *')
+
+// Every 5 minutes
+await scheduler.schedule('events', '*/5 * * * *')
+
+// Every day at 2:30 AM
+await scheduler.schedule('events', '30 2 * * *')
+
+// Every Monday at 9:00 AM
+await scheduler.schedule('events', '0 9 * * 1')
+```
+
+## API Reference
+
+### SnapshotScheduler
+
+#### `initialize()`
+
+Initialize the scheduler and load persisted state.
+
+```typescript
+await scheduler.initialize()
+```
+
+#### `schedule(table, cron, retentionPolicy?)`
+
+Schedule automatic snapshots for a table.
+
+```typescript
+const schedule = await scheduler.schedule(
+  'events',
+  '0 * * * *', // Hourly
+  {
+    maxAgeDays: 30,
+    minSnapshotsToRetain: 5,
+  }
+)
+```
+
+**Parameters:**
+- `table` - Table identifier
+- `cron` - Cron expression
+- `retentionPolicy` - Optional retention policy
+
+**Returns:** `SnapshotSchedule`
+
+#### `unschedule(table)`
+
+Remove scheduled snapshots for a table.
+
+```typescript
+const deleted = await scheduler.unschedule('events')
+```
+
+**Returns:** `boolean` - Whether the schedule was deleted
+
+#### `toggleSchedule(table, enabled)`
+
+Enable or disable a schedule.
+
+```typescript
+await scheduler.toggleSchedule('events', false) // Disable
+await scheduler.toggleSchedule('events', true)  // Enable
+```
+
+#### `getSchedule(table)`
+
+Get schedule for a specific table.
+
+```typescript
+const schedule = scheduler.getSchedule('events')
+if (schedule) {
+  console.log(`Next run: ${schedule.nextRun}`)
+}
+```
+
+#### `listScheduled()`
+
+List all scheduled snapshots.
+
+```typescript
+const schedules = scheduler.listScheduled()
+for (const schedule of schedules) {
+  console.log(`${schedule.table}: ${schedule.cron}`)
+}
+```
+
+### Write Thresholds
+
+#### `setThreshold(table, writeCount)`
+
+Set write count threshold for automatic snapshots.
+
+```typescript
+await scheduler.setThreshold('events', 10000)
+```
+
+#### `removeThreshold(table)`
+
+Remove write threshold for a table.
+
+```typescript
+await scheduler.removeThreshold('events')
+```
+
+#### `incrementWriteCount(table, count?)`
+
+Increment write count and trigger snapshot if threshold reached.
+
+```typescript
+// After inserting 100 rows
+const shouldSnapshot = await scheduler.incrementWriteCount('events', 100)
+
+if (shouldSnapshot) {
+  console.log('Threshold reached, snapshot created')
+}
+```
+
+#### `getThreshold(table)`
+
+Get write threshold for a table.
+
+```typescript
+const threshold = scheduler.getThreshold('events')
+if (threshold) {
+  console.log(`${threshold.currentCount} / ${threshold.writeCount}`)
+}
+```
+
+#### `listThresholds()`
+
+List all write thresholds.
+
+```typescript
+const thresholds = scheduler.listThresholds()
+```
+
+### Retention Enforcement
+
+#### `enforceRetention(table, policy)`
+
+Manually enforce retention policy for a table.
+
+```typescript
+const result = await scheduler.enforceRetention('events', {
+  maxAgeDays: 30,
+  minSnapshotsToRetain: 5,
+  maxSnapshotsToRetain: 100,
+})
+
+console.log(`Expired ${result.snapshotsExpired} snapshots`)
+console.log(`Freed ${result.metadataBytesFreed} bytes`)
+```
+
+### Processing
+
+#### `processDueSnapshots()`
+
+Process all due snapshots (called by alarm or cron trigger).
+
+```typescript
+const result = await scheduler.processDueSnapshots()
+
+console.log(`Created ${result.snapshotsCreated.length} snapshots`)
+console.log(`Next alarm: ${result.nextAlarmAt}`)
+```
+
+### Statistics
+
+#### `getStats()`
+
+Get scheduler statistics.
+
+```typescript
+const stats = scheduler.getStats()
+
+console.log('Scheduled snapshots:', stats.scheduledSnapshotsCreated)
+console.log('Threshold snapshots:', stats.thresholdSnapshotsCreated)
+console.log('Retention enforcements:', stats.retentionEnforcements)
+console.log('Last run:', new Date(stats.lastRunAt))
+```
+
+#### `resetStats()`
+
+Reset scheduler statistics.
+
+```typescript
+await scheduler.resetStats()
+```
+
+## Cloudflare Integration
+
+### Durable Object Alarms
+
+The scheduler automatically sets Durable Object alarms for the next scheduled snapshot.
+
+```typescript
+export class IcebergSchedulerDO extends DurableObject {
+  private scheduler: SnapshotScheduler | null = null
+
+  async alarm() {
+    if (!this.scheduler) {
+      this.scheduler = createSnapshotScheduler({
+        bucket: this.env.ICEBERG_BUCKET,
+        tablePath: 'iceberg',
+        state: this.ctx.state,
+      })
+      await this.scheduler.initialize()
+    }
+
+    await this.scheduler.processDueSnapshots()
+  }
+}
+```
+
+### Cron Triggers
+
+Configure cron triggers in `wrangler.toml`:
+
+```toml
+[triggers]
+crons = ["0 * * * *"]  # Every hour
+```
+
+Handle in your worker:
+
+```typescript
+export default {
+  async scheduled(event: ScheduledEvent, env: Env, ctx: ExecutionContext) {
+    const id = env.ICEBERG_SCHEDULER.idFromName('global')
+    const stub = env.ICEBERG_SCHEDULER.get(id)
+    ctx.waitUntil(stub.fetch('https://scheduler/process'))
+  }
+}
+```
+
+Or use the helper function:
+
+```typescript
+import { handleScheduledSnapshots } from '@dotdo/postgres/iceberg'
+
+export default {
+  async scheduled(event: ScheduledEvent, env: Env, ctx: ExecutionContext) {
+    await handleScheduledSnapshots(event, env, ctx)
+  }
+}
+```
+
+## Configuration
+
+### SnapshotSchedulerConfig
+
+```typescript
+interface SnapshotSchedulerConfig {
+  /** R2 bucket for Iceberg storage */
+  bucket: R2Bucket
+
+  /** Base path for tables in R2 */
+  tablePath: string
+
+  /** Durable Object state for persistence (optional) */
+  state?: DurableObjectState
+
+  /** Check interval for alarm scheduling (default: 60000 = 1 minute) */
+  checkIntervalMs?: number
+
+  /** Default retention policy for new schedules */
+  defaultRetention?: RetentionConfig
+}
+```
+
+### RetentionConfig
+
+```typescript
+interface RetentionConfig {
+  /** Maximum age of snapshots to keep (in days) */
+  maxAgeDays?: number
+
+  /** Minimum number of snapshots to retain regardless of age */
+  minSnapshotsToRetain?: number
+
+  /** Maximum number of snapshots to keep */
+  maxSnapshotsToRetain?: number
+
+  /** Specific snapshot IDs to always retain */
+  retainSnapshotIds?: number[]
+}
+```
+
+## Best Practices
+
+### 1. Use Durable Objects for Coordination
+
+When running multiple Workers, use a Durable Object to coordinate scheduling:
+
+```typescript
+// All workers forward to the same DO
+const id = env.ICEBERG_SCHEDULER.idFromName('global')
+const stub = env.ICEBERG_SCHEDULER.get(id)
+```
+
+### 2. Set Appropriate Retention Policies
+
+Balance storage costs with time-travel requirements:
+
+```typescript
+await scheduler.schedule('events', '0 * * * *', {
+  maxAgeDays: 7,        // Keep snapshots for a week
+  minSnapshotsToRetain: 24, // Keep at least last 24 hours
+})
+```
+
+### 3. Monitor Write Thresholds
+
+For high-volume tables, use write thresholds instead of time-based schedules:
+
+```typescript
+// Create snapshot after 1 million writes
+await scheduler.setThreshold('high_volume_table', 1000000)
+```
+
+### 4. Combine Schedules and Thresholds
+
+Use both for comprehensive snapshot coverage:
+
+```typescript
+// Hourly snapshots
+await scheduler.schedule('events', '0 * * * *')
+
+// Also snapshot after 100k writes
+await scheduler.setThreshold('events', 100000)
+```
+
+### 5. Test Cron Expressions
+
+Use the `parseNextRun` helper to verify cron expressions:
+
+```typescript
+import { parseNextRun } from '@dotdo/postgres/iceberg'
+
+const nextRun = parseNextRun('0 * * * *')
+console.log('Next run:', nextRun)
+```
+
+## Examples
+
+### Complete Scheduler Setup
+
+```typescript
+import {
+  createSnapshotScheduler,
+  type SnapshotSchedulerConfig,
+} from '@dotdo/postgres/iceberg'
+
+// Create scheduler
+const scheduler = createSnapshotScheduler({
+  bucket: env.ICEBERG_BUCKET,
+  tablePath: 'iceberg',
+  state: durableObjectState,
+  defaultRetention: {
+    maxAgeDays: 30,
+    minSnapshotsToRetain: 5,
+  },
+})
+
+await scheduler.initialize()
+
+// Schedule different tables
+await scheduler.schedule('events', '0 * * * *')      // Hourly
+await scheduler.schedule('analytics', '0 0 * * *')   // Daily
+await scheduler.schedule('backups', '0 0 * * 0')     // Weekly
+
+// Set write thresholds
+await scheduler.setThreshold('high_volume', 1000000)
+await scheduler.setThreshold('medium_volume', 100000)
+
+// Get statistics
+const stats = scheduler.getStats()
+console.log('Scheduler statistics:', stats)
+```
+
+### Monitoring Scheduled Snapshots
+
+```typescript
+// List all schedules
+const schedules = scheduler.listScheduled()
+
+for (const schedule of schedules) {
+  console.log(`Table: ${schedule.table}`)
+  console.log(`  Cron: ${schedule.cron}`)
+  console.log(`  Enabled: ${schedule.enabled}`)
+  console.log(`  Next run: ${schedule.nextRun}`)
+  console.log(`  Last run: ${schedule.lastRun ?? 'Never'}`)
+
+  if (schedule.retentionPolicy) {
+    console.log(`  Retention: ${schedule.retentionPolicy.maxAgeDays} days`)
+  }
+}
+```
+
+### Write Count Integration
+
+```typescript
+// In your write path
+async function insertData(table: string, rows: any[]) {
+  // Insert data
+  await insertRows(table, rows)
+
+  // Increment write count
+  const shouldSnapshot = await scheduler.incrementWriteCount(
+    table,
+    rows.length
+  )
+
+  if (shouldSnapshot) {
+    console.log(`Threshold reached for ${table}, snapshot created`)
+  }
+}
+```
+
+## Troubleshooting
+
+### Schedules Not Running
+
+1. Check alarm is set:
+   ```typescript
+   const alarm = await state.storage.getAlarm()
+   console.log('Next alarm:', alarm ? new Date(alarm) : 'Not set')
+   ```
+
+2. Verify schedule is enabled:
+   ```typescript
+   const schedule = scheduler.getSchedule('events')
+   console.log('Enabled:', schedule?.enabled)
+   ```
+
+3. Check for errors in alarm handler:
+   ```typescript
+   async alarm() {
+     try {
+       await scheduler.processDueSnapshots()
+     } catch (error) {
+       console.error('Alarm error:', error)
+     }
+   }
+   ```
+
+### High Storage Costs
+
+Adjust retention policies:
+
+```typescript
+await scheduler.enforceRetention('events', {
+  maxAgeDays: 7,         // Reduce from 30 days
+  maxSnapshotsToRetain: 50, // Set maximum limit
+})
+```
+
+### Performance Issues
+
+1. Reduce check interval:
+   ```typescript
+   createSnapshotScheduler({
+     bucket: env.ICEBERG_BUCKET,
+     tablePath: 'iceberg',
+     checkIntervalMs: 300000, // 5 minutes instead of 1 minute
+   })
+   ```
+
+2. Use write thresholds for high-volume tables:
+   ```typescript
+   // Instead of frequent time-based snapshots
+   await scheduler.setThreshold('high_volume', 1000000)
+   ```
+
+## See Also
+
+- [Snapshot Manager](./snapshot-manager.ts) - Snapshot lifecycle and compaction
+- [Time Travel](./time-travel.ts) - Query historical snapshots
+- [Iceberg Catalog](./catalog.ts) - Table catalog management
+- [Example DO Integration](./scheduler-do-example.ts) - Complete Durable Object example