reqon-dsl 0.2.0

package/docusaurus/docs/http/incremental-sync.md

@@ -0,0 +1,394 @@
+---
+sidebar_position: 4
+---
+
+# Incremental Sync
+
+Incremental sync allows you to fetch only changes since the last run, reducing API calls and improving performance.
+
+## Basic Usage
+
+```vague
+get "/items" {
+  since: lastSync
+}
+```
+
+This automatically:
+1. Checks when the last successful sync occurred
+2. Adds a timestamp parameter to the request
+3. Updates the checkpoint after successful completion
+
+## How It Works
+
+### First Run
+
+On the first run, no `since` parameter is added:
+
+```
+GET /items
+```
+
+### Subsequent Runs
+
+On subsequent runs, the last sync timestamp is used:
+
+```
+GET /items?modified_since=2024-01-20T10:30:00Z
+```
+
+### Checkpoint Storage
+
+Checkpoints are stored in `.vague-data/` by default:
+
+```
+.vague-data/
+├── sync-checkpoints.json
+└── stores/
+```
+
+## Configuration
+
+### Custom Parameter Name
+
+Specify the API's expected parameter:
+
+```vague
+get "/items" {
+  since: lastSync,
+  sinceParam: "updatedAfter"
+}
+```
+
+Generates: `?updatedAfter=2024-01-20T10:30:00Z`
+
+### Date Format
+
+Customize the date format:
+
+```vague
+get "/items" {
+  since: lastSync,
+  sinceFormat: "YYYY-MM-DD"
+}
+```
+
+Common formats:
+- `"ISO"` - ISO 8601 (default): `2024-01-20T10:30:00Z`
+- `"YYYY-MM-DD"` - Date only: `2024-01-20`
+- `"timestamp"` - Unix timestamp: `1705748400`
+- `"epoch"` - Unix epoch milliseconds: `1705748400000`
+
+### Custom Checkpoint Key
+
+Override the automatic checkpoint key:
+
+```vague
+get "/items" {
+  since: lastSync,
+  syncKey: "items-main-sync"
+}
+```
+
+## Combining with Pagination
+
+```vague
+get "/items" {
+  paginate: offset(offset, 100),
+  until: length(response.items) == 0,
+  since: lastSync
+}
+```
+
+The `since` parameter is added to each paginated request.
+
+## Combining with Filters
+
+```vague
+get "/items" {
+  params: {
+    status: "active",
+    type: "order"
+  },
+  since: lastSync
+}
+```
+
+Generates: `?status=active&type=order&modified_since=2024-01-20T10:30:00Z`
+
+## Handling Updates
+
+Use upsert mode for incremental updates:
+
+```vague
+action IncrementalSync {
+  get "/items" {
+    paginate: offset(offset, 100),
+    until: length(response.items) == 0,
+    since: lastSync
+  }
+
+  for item in response.items {
+    store item -> items { key: .id, upsert: true }
+  }
+}
+```
+
+## Per-Source Checkpoints
+
+Different sources maintain separate checkpoints:
+
+```vague
+mission MultiSourceSync {
+  source Xero { auth: oauth2, base: "https://api.xero.com" }
+  source QuickBooks { auth: oauth2, base: "https://quickbooks.api.com" }
+
+  action SyncXero {
+    get Xero "/invoices" { since: lastSync }
+    // Uses Xero-specific checkpoint
+  }
+
+  action SyncQuickBooks {
+    get QuickBooks "/invoices" { since: lastSync }
+    // Uses QuickBooks-specific checkpoint
+  }
+}
+```
+
+## Per-Endpoint Checkpoints
+
+Each endpoint maintains its own checkpoint:
+
+```vague
+action SyncAll {
+  get "/customers" { since: lastSync }
+  // Checkpoint: source-/customers
+
+  get "/orders" { since: lastSync }
+  // Checkpoint: source-/orders
+
+  get "/products" { since: lastSync }
+  // Checkpoint: source-/products
+}
+```
+
+## Resetting Checkpoints
+
+### Via CLI
+
+```bash
+# Reset all checkpoints
+rm -rf .vague-data/sync-checkpoints.json
+
+# Then run a full sync
+reqon sync.vague
+```
+
+### Programmatically
+
+```typescript
+import { execute, clearSyncCheckpoints } from 'reqon';
+
+// Clear all checkpoints
+await clearSyncCheckpoints();
+
+// Or clear specific checkpoint
+await clearSyncCheckpoint('source-/items');
+```
+
+## Full Sync vs Incremental
+
+### Force Full Sync
+
+Sometimes you need a full resync:
+
+```vague
+action FullSync {
+  get "/items"  // No since option = full sync
+  store response -> items { key: .id, upsert: true }
+}
+
+action IncrementalSync {
+  get "/items" { since: lastSync }
+  store response -> items { key: .id, upsert: true }
+}
+
+// Use conditional in pipeline
+run IncrementalSync  // Default: incremental
+// Or: run FullSync when needed
+```
+
+### Conditional Sync
+
+```vague
+action SmartSync {
+  get "/status"
+
+  match response {
+    { needsFullSync: true } -> {
+      get "/items"
+      store response -> items { key: .id }
+    },
+    _ -> {
+      get "/items" { since: lastSync }
+      store response -> items { key: .id, upsert: true }
+    }
+  }
+}
+```
+
+## Handling Deletions
+
+Incremental sync doesn't automatically handle deleted items. Handle this based on your API:
+
+### Soft Deletes
+
+```vague
+get "/items" {
+  params: { includeDeleted: true },
+  since: lastSync
+}
+
+for item in response.items {
+  match item {
+    { deleted: true } -> {
+      // Remove from local store
+      delete items[item.id]
+    },
+    _ -> store item -> items { key: .id, upsert: true }
+  }
+}
+```
+
+### Deletion Endpoint
+
+```vague
+action SyncItems {
+  get "/items" { since: lastSync }
+  store response -> items { key: .id, upsert: true }
+}
+
+action SyncDeletions {
+  get "/items/deleted" { since: lastSync }
+
+  for deletion in response.deletions {
+    delete items[deletion.id]
+  }
+}
+
+run [SyncItems, SyncDeletions]
+```
+
+## Best Practices
+
+### Always Use Upsert
+
+```vague
+// Good: handles both new and updated items
+store item -> items { key: .id, upsert: true }
+
+// Risky: may fail on duplicates
+store item -> items { key: .id }
+```
+
+### Handle Empty Responses
+
+```vague
+get "/items" { since: lastSync }
+
+match response {
+  { items: [] } -> {
+    // No updates since last sync - this is fine
+    continue
+  },
+  _ -> {
+    for item in response.items {
+      store item -> items { key: .id, upsert: true }
+    }
+  }
+}
+```
+
+### Log Sync Progress
+
+```vague
+action IncrementalSync {
+  get "/items" { since: lastSync }
+
+  store {
+    timestamp: now(),
+    itemCount: length(response.items),
+    type: "incremental"
+  } -> syncLogs
+
+  for item in response.items {
+    store item -> items { key: .id, upsert: true }
+  }
+}
+```
+
+### Schedule Regular Syncs
+
+```vague
+mission RegularSync {
+  schedule: every 15 minutes
+
+  action Sync {
+    get "/items" { since: lastSync }
+    store response -> items { key: .id, upsert: true }
+  }
+
+  run Sync
+}
+```
+
+## Troubleshooting
+
+### Checkpoint Not Updating
+
+Checkpoints only update on successful completion. Check for errors:
+
+```vague
+action DebugSync {
+  get "/items" { since: lastSync }
+
+  match response {
+    { error: e } -> {
+      store { error: e, timestamp: now() } -> syncErrors
+      abort e
+    },
+    _ -> continue
+  }
+
+  for item in response.items {
+    store item -> items { key: .id, upsert: true }
+  }
+}
+```
+
+### Wrong Date Format
+
+Match your API's expected format:
+
+```vague
+// For APIs expecting ISO 8601
+get "/items" { since: lastSync }
+
+// For APIs expecting Unix timestamp
+get "/items" { since: lastSync, sinceFormat: "timestamp" }
+
+// For APIs expecting date only
+get "/items" { since: lastSync, sinceFormat: "YYYY-MM-DD" }
+```
+
+### Missing Updates
+
+Ensure your API uses the same field for filtering:
+
+```vague
+// If API uses "updatedAt" field
+get "/items" { since: lastSync, sinceParam: "updatedAt" }
+
+// If API uses "modifiedSince" parameter
+get "/items" { since: lastSync, sinceParam: "modifiedSince" }
+```