reqon-dsl 0.2.0

package/examples/README.md

@@ -0,0 +1,112 @@
+# Reqon Examples
+
+This directory contains examples demonstrating Reqon's features for declarative data pipelines.
+
+## Examples Overview
+
+| Example | Description | Key Features |
+|---------|-------------|--------------|
+| [jsonplaceholder](./jsonplaceholder/) | Basic public API sync | `auth: none`, fetch, map, for loops |
+| [petstore](./petstore/) | OpenAPI spec integration | OAS operationId, cursor pagination |
+| [xero](./xero/) | OAuth2 invoice sync | OAuth2, hydration, **match steps**, **flow control** |
+| [github-sync](./github-sync/) | Multi-file mission | **Folder structure**, **parallel execution**, schema matching |
+| [error-handling](./error-handling/) | Comprehensive error handling | **All flow control directives**, dead letter queues |
+| [temporal-comparison](./temporal-comparison/) | E-commerce reconciliation | Multi-source, **parallel execution**, rate limiting |
+
+## Feature Index
+
+### Multi-File Missions
+Organize large missions into folders with separate action files:
+```
+github-sync/
+├── mission.vague       # Sources, stores, schemas, pipeline
+├── fetch-issues.vague  # Action file
+├── fetch-prs.vague     # Action file
+└── normalize.vague     # Action file
+```
+See: [github-sync](./github-sync/)
+
+### Parallel Execution
+Run multiple actions concurrently:
+```vague
+run [FetchOrders, FetchPayments, FetchShipments] then Reconcile
+```
+See: [github-sync](./github-sync/), [temporal-comparison](./temporal-comparison/)
+
+### Schema Overloading with Match Steps
+Handle different API response types declaratively:
+```vague
+match response {
+  SuccessSchema -> { store response -> cache },
+  RateLimitError -> retry { maxAttempts: 5 },
+  AuthError -> jump RefreshToken then retry,
+  _ -> abort "Unexpected response"
+}
+```
+See: [xero](./xero/), [error-handling](./error-handling/)
+
+### Flow Control Directives
+Six directives for controlling execution flow:
+
+| Directive | Description | Example |
+|-----------|-------------|---------|
+| `continue` | Proceed to next step | `Schema -> continue` |
+| `skip` | Skip remaining steps | `Schema -> skip` |
+| `abort` | Halt mission | `Schema -> abort "Error"` |
+| `retry` | Retry with backoff | `Schema -> retry { maxAttempts: 5 }` |
+| `queue` | Send to dead letter queue | `Schema -> queue dlq` |
+| `jump` | Execute action, then continue | `Schema -> jump Refresh then retry` |
+
+See: [error-handling](./error-handling/)
+
+### Authentication Types
+```vague
+source API { auth: none }           # Public API
+source API { auth: bearer }         # Bearer token
+source API { auth: oauth2 }         # OAuth2
+source API { auth: basic }          # Basic auth
+source API { auth: api_key }        # API key
+```
+
+### Pagination Strategies
+```vague
+paginate: offset(page, 100)                    # Offset pagination
+paginate: page(page, 100)                      # Page number pagination
+paginate: cursor(cursor, 100, "nextCursor")   # Cursor pagination
+```
+
+## Running Examples
+
+```bash
+# Build first
+npm run build
+
+# Run any example
+node dist/cli.js examples/<example>/<file>.vague --verbose
+
+# Run multi-file mission (folder)
+node dist/cli.js examples/github-sync --verbose
+
+# Dry run (no actual API calls)
+node dist/cli.js examples/xero/invoices.vague --dry-run
+
+# With credentials
+node dist/cli.js examples/xero/invoices.vague --auth credentials.json
+```
+
+## Credentials Format
+
+Create a `credentials.json` file:
+```json
+{
+  "SourceName": {
+    "type": "bearer",
+    "token": "your-token"
+  },
+  "OAuthSource": {
+    "type": "oauth2",
+    "accessToken": "your-access-token",
+    "refreshToken": "your-refresh-token"
+  }
+}
+```

package/examples/error-handling/README.md

@@ -0,0 +1,150 @@
+# Error Handling Example
+
+Demonstrates comprehensive error handling with schema matching and all flow control directives.
+
+## What it does
+
+1. **FetchPendingPayments**: Fetches payments with retry and auth refresh
+2. **CheckFraudRisk**: Checks each payment for fraud, routing high-risk to review queue
+3. **ProcessPayments**: Captures approved payments with full error handling
+4. **GenerateReport**: Creates a summary of processing results
+
+## Run
+
+```bash
+node dist/cli.js examples/error-handling/payment-processor.vague --auth credentials.json --verbose
+```
+
+## Flow Control Directives
+
+This example demonstrates all six flow control directives:
+
+### 1. `continue` - Proceed to next step
+```vague
+match response {
+  FraudWarning where .risk_level == "low" -> continue
+}
+```
+When the condition matches, continue with the next steps in the action.
+
+### 2. `skip` - Skip remaining steps in current iteration
+```vague
+match response {
+  _ -> skip  // Unknown response - skip this payment, move to next
+}
+```
+Skips remaining steps for the current item in a `for` loop but continues with the next item.
+
+### 3. `abort "message"` - Halt mission with error
+```vague
+match response {
+  ServerError -> abort "Payment gateway unavailable"
+}
+```
+Immediately stops the entire mission and reports the error.
+
+### 4. `retry { config }` - Retry with backoff
+```vague
+match response {
+  RateLimitError -> retry {
+    maxAttempts: 5,
+    backoff: exponential,    // or: linear, constant
+    initialDelay: 5000,
+    maxDelay: 120000
+  }
+}
+```
+Retries the current fetch with configurable backoff strategy.
+
+### 5. `queue target` - Send to dead-letter queue
+```vague
+match response {
+  ServerError -> queue dead_letter_queue
+}
+```
+Parks the current item for later processing or manual review.
+
+### 6. `jump Action then retry` - Execute action, then continue
+```vague
+match response {
+  AuthenticationError -> jump RefreshAuth then retry
+}
+```
+Jumps to another action (e.g., to refresh auth), then retries the original request.
+
+## Schema Matching Patterns
+
+### Simple Schema Match
+```vague
+match response {
+  PaymentSuccess -> { store response -> payments }
+}
+```
+
+### Schema with Guard Clause
+```vague
+match response {
+  FraudWarning where .risk_level == "high" -> queue fraud_review
+}
+```
+
+### Array Schema Match
+```vague
+match response {
+  [PaymentSuccess] -> { store response -> payments { key: .id } }
+}
+```
+
+### Wildcard (Catch-All)
+```vague
+match response {
+  SuccessSchema -> continue,
+  ErrorSchema -> abort "Error occurred",
+  _ -> skip  // Handle any other response
+}
+```
+
+## Error Handling Patterns
+
+### 1. Retry with Auth Refresh
+```vague
+AuthenticationError -> jump RefreshAuth then retry
+```
+
+### 2. Progressive Backoff
+```vague
+RateLimitError -> retry {
+  maxAttempts: 5,
+  backoff: exponential,
+  initialDelay: 1000,
+  maxDelay: 60000
+}
+```
+
+### 3. Dead Letter Queue for Failures
+```vague
+ServerError -> queue dead_letter_queue
+```
+
+### 4. Conditional Processing
+```vague
+match response {
+  FraudWarning where .risk_level == "low" -> continue,
+  FraudWarning where .risk_level == "medium" -> {
+    store { ... } -> review_queue
+  },
+  FraudWarning where .risk_level == "high" -> {
+    store { ... } -> fraud_queue
+  }
+}
+```
+
+## Features demonstrated
+
+- All 6 flow control directives
+- Schema matching with guard clauses
+- Retry with exponential/linear/constant backoff
+- Token refresh via `jump...then retry`
+- Dead letter queue pattern
+- Conditional routing based on response content
+- Error logging and audit trails

package/examples/error-handling/payment-processor.vague

@@ -0,0 +1,287 @@
+// Payment Processing Pipeline with Comprehensive Error Handling
+// Demonstrates all flow control directives: continue, skip, abort, retry, queue, jump
+
+mission ProcessPayments {
+
+  // ============================================================
+  // SOURCES
+  // ============================================================
+
+  source PaymentGateway {
+    auth: bearer,
+    base: "https://api.paymentgateway.com/v1",
+    rateLimit: {
+      strategy: "pause",
+      maxWait: 60,
+      fallbackRpm: 100
+    }
+  }
+
+  source FraudDetection {
+    auth: api_key,
+    base: "https://fraud.example.com/api"
+  }
+
+  // ============================================================
+  // STORES
+  // ============================================================
+
+  store pending_payments: memory("pending")
+  store processed_payments: sql("payments")
+  store failed_payments: memory("failed")
+  store fraud_review_queue: memory("fraud_review")
+  store dead_letter_queue: memory("dlq")
+
+  // ============================================================
+  // SCHEMAS - Response Types
+  // ============================================================
+
+  schema PaymentSuccess {
+    id: string,
+    status: string,
+    amount: decimal,
+    currency: string,
+    captured_at: date
+  }
+
+  schema PaymentPending {
+    id: string,
+    status: string,
+    requires_action: boolean
+  }
+
+  schema RateLimitError {
+    error: string,
+    retry_after: int
+  }
+
+  schema AuthenticationError {
+    error: string,
+    code: string
+  }
+
+  schema ValidationError {
+    error: string,
+    field: string,
+    message: string
+  }
+
+  schema FraudWarning {
+    risk_score: decimal,
+    risk_level: string,
+    recommendation: string
+  }
+
+  schema ServerError {
+    error: string,
+    request_id: string
+  }
+
+  // ============================================================
+  // ACTION: Fetch Pending Payments
+  // ============================================================
+
+  action FetchPendingPayments {
+    get "/payments" {
+      body: { "status": "pending" },
+      retry: {
+        maxAttempts: 3,
+        backoff: "exponential",
+        initialDelay: 1000
+      }
+    }
+
+    match response {
+      // Success - store and continue
+      [PaymentSuccess] -> {
+        store response -> pending_payments { key: .id }
+      },
+
+      // Rate limited - retry with backoff
+      RateLimitError -> retry {
+        maxAttempts: 5,
+        backoff: exponential,
+        initialDelay: 5000,
+        maxDelay: 120000
+      },
+
+      // Auth error - refresh token and retry
+      AuthenticationError -> jump RefreshAuth then retry,
+
+      // Server error - abort the mission
+      ServerError -> abort "Payment gateway unavailable",
+
+      // Unexpected response - log and continue with empty
+      _ -> {
+        store {
+          error: "Unexpected response fetching payments",
+          response: response,
+          timestamp: now()
+        } -> failed_payments { key: now() }
+      }
+    }
+  }
+
+  // ============================================================
+  // ACTION: Refresh Authentication (called via jump)
+  // ============================================================
+
+  action RefreshAuth {
+    post "/auth/refresh" {
+      source: PaymentGateway,
+      body: {
+        "refresh_token": env("PAYMENT_REFRESH_TOKEN")
+      }
+    }
+
+    match response {
+      _ where .access_token != null -> continue,
+      _ -> abort "Failed to refresh authentication"
+    }
+  }
+
+  // ============================================================
+  // ACTION: Check Fraud Risk
+  // ============================================================
+
+  action CheckFraudRisk {
+    for payment in pending_payments {
+      post "/check" {
+        source: FraudDetection,
+        body: {
+          "payment_id": payment.id,
+          "amount": payment.amount,
+          "currency": payment.currency
+        }
+      }
+
+      match response {
+        // Low risk - continue processing
+        FraudWarning where .risk_level == "low" -> continue,
+
+        // Medium risk - queue for human review, skip this payment
+        FraudWarning where .risk_level == "medium" -> {
+          store {
+            payment_id: payment.id,
+            risk_score: response.risk_score,
+            reason: "Medium risk - requires review"
+          } -> fraud_review_queue { key: payment.id }
+        },
+
+        // High risk - queue and skip
+        FraudWarning where .risk_level == "high" -> {
+          store {
+            payment_id: payment.id,
+            risk_score: response.risk_score,
+            reason: "High fraud risk detected"
+          } -> fraud_review_queue { key: payment.id }
+        },
+
+        // Fraud service error - queue to DLQ for later processing
+        ServerError -> queue dead_letter_queue,
+
+        // Rate limited - retry
+        RateLimitError -> retry {
+          maxAttempts: 3,
+          backoff: constant,
+          initialDelay: 10000
+        },
+
+        // Unknown - skip but log
+        _ -> skip
+      }
+    }
+  }
+
+  // ============================================================
+  // ACTION: Process Approved Payments
+  // ============================================================
+
+  action ProcessPayments {
+    // Only process payments not in fraud review
+    for payment in pending_payments where not exists(fraud_review_queue[payment.id]) {
+      post "/payments/{payment.id}/capture" {
+        source: PaymentGateway,
+        body: {
+          "idempotency_key": payment.id + "_capture"
+        }
+      }
+
+      match response {
+        // Successfully captured
+        PaymentSuccess -> {
+          store response -> processed_payments {
+            key: .id,
+            upsert: true
+          }
+        },
+
+        // Payment requires additional action (3DS, etc)
+        PaymentPending where .requires_action == true -> {
+          store {
+            payment_id: payment.id,
+            reason: "Requires customer action",
+            status: "pending_action"
+          } -> failed_payments { key: payment.id }
+        },
+
+        // Validation error - payment data issue
+        ValidationError -> {
+          store {
+            payment_id: payment.id,
+            error: response.message,
+            field: response.field,
+            status: "invalid"
+          } -> failed_payments { key: payment.id }
+        },
+
+        // Rate limited
+        RateLimitError -> retry {
+          maxAttempts: 3,
+          backoff: exponential,
+          initialDelay: 2000
+        },
+
+        // Auth expired during processing
+        AuthenticationError -> jump RefreshAuth then retry,
+
+        // Server error - queue for retry later
+        ServerError -> queue dead_letter_queue,
+
+        // Catch-all - queue to DLQ
+        _ -> queue dead_letter_queue
+      }
+    }
+  }
+
+  // ============================================================
+  // ACTION: Generate Report
+  // ============================================================
+
+  action GenerateReport {
+    map {
+      processed: length(processed_payments),
+      failed: length(failed_payments),
+      fraud_flagged: length(fraud_review_queue),
+      queued_for_retry: length(dead_letter_queue),
+      completed_at: now()
+    } -> report
+
+    validate report {
+      assume .processed >= 0
+    }
+
+    store response -> processed_payments {
+      key: "report_" + now()
+    }
+  }
+
+  // ============================================================
+  // PIPELINE
+  // ============================================================
+
+  run FetchPendingPayments
+    then CheckFraudRisk
+    then ProcessPayments
+    then GenerateReport
+}

package/examples/github-sync/README.md

@@ -0,0 +1,74 @@
+# GitHub Sync Example
+
+Demonstrates multi-file missions, parallel execution, and error handling with flow control.
+
+## Folder Structure
+
+This mission is organized as a folder with separate action files:
+
+```
+github-sync/
+├── mission.vague       # Main file: sources, stores, schemas, pipeline
+├── fetch-issues.vague  # Fetches issues with pagination and error handling
+├── fetch-prs.vague     # Fetches pull requests with error handling
+├── normalize.vague     # Normalizes to unified schema
+└── README.md
+```
+
+## What it does
+
+1. **FetchIssues** and **FetchPRs** run in **parallel** to fetch data concurrently
+2. **Normalize** runs after both complete, unifying issues and PRs into `work_items`
+
+## Run
+
+```bash
+# Run from folder path
+node dist/cli.js examples/github-sync --auth credentials.json --verbose
+
+# Environment variables needed
+export GITHUB_OWNER=your-org
+export GITHUB_REPO=your-repo
+```
+
+Requires a `credentials.json`:
+```json
+{
+  "GitHub": {
+    "type": "bearer",
+    "token": "ghp_your_github_token"
+  }
+}
+```
+
+## Features demonstrated
+
+### Multi-file Missions
+- `mission.vague` contains sources, stores, schemas, and pipeline
+- Action files (`fetch-issues.vague`, etc.) are automatically discovered and merged
+- Actions can reference stores and schemas defined in the root file
+
+### Parallel Execution
+```vague
+run [FetchIssues, FetchPRs] then Normalize
+```
+- `[FetchIssues, FetchPRs]` runs both actions concurrently
+- `then Normalize` waits for both to complete before running
+
+### Schema Overloading with Match Steps
+```vague
+match response {
+  [GitHubIssue] -> { store response -> issues_raw },
+  RateLimitError -> retry { maxAttempts: 5 },
+  NotFoundError -> abort "Repository not found",
+  _ -> skip
+}
+```
+
+### Flow Control Directives
+- `continue` - Proceed to next step
+- `skip` - Skip remaining steps in current action
+- `abort "message"` - Halt mission with error
+- `retry { ... }` - Retry with backoff configuration
+- `queue target` - Send to dead-letter queue
+- `jump ActionName then retry` - Execute another action, then retry

package/examples/github-sync/fetch-issues.vague

@@ -0,0 +1,47 @@
+// Fetch GitHub Issues with error handling
+// External action file - automatically merged into mission
+
+action FetchIssues {
+  get "/repos/{GITHUB_OWNER}/{GITHUB_REPO}/issues" {
+    source: GitHub,
+    body: {
+      "state": "all",
+      "per_page": 100
+    },
+    paginate: page(page, 100),
+    until: length(response) == 0,
+    retry: {
+      maxAttempts: 3,
+      backoff: "exponential",
+      initialDelay: 1000
+    }
+  }
+
+  // Handle different response types with schema matching
+  match response {
+    // Success - continue processing
+    [GitHubIssue] -> {
+      store response -> issues_raw { key: .id }
+    },
+
+    // Rate limited - retry with backoff
+    RateLimitError -> retry {
+      maxAttempts: 5,
+      backoff: exponential,
+      initialDelay: 60000,
+      maxDelay: 300000
+    },
+
+    // Repo not found - abort mission
+    NotFoundError -> abort "Repository not found - check GITHUB_OWNER and GITHUB_REPO",
+
+    // Unexpected response - log and skip
+    _ -> {
+      store {
+        error: "Unexpected response in FetchIssues",
+        response: response,
+        timestamp: now()
+      } -> errors { key: now() }
+    }
+  }
+}

package/examples/github-sync/fetch-prs.vague

@@ -0,0 +1,40 @@
+// Fetch GitHub Pull Requests with error handling
+// External action file - automatically merged into mission
+
+action FetchPRs {
+  get "/repos/{GITHUB_OWNER}/{GITHUB_REPO}/pulls" {
+    source: GitHub,
+    body: {
+      "state": "all",
+      "per_page": 100
+    },
+    paginate: page(page, 100),
+    until: length(response) == 0,
+    retry: {
+      maxAttempts: 3,
+      backoff: "exponential",
+      initialDelay: 1000
+    }
+  }
+
+  // Handle different response types
+  match response {
+    // Success - store PRs
+    [GitHubPR] -> {
+      store response -> prs_raw { key: .id }
+    },
+
+    // Rate limited - retry
+    RateLimitError -> retry {
+      maxAttempts: 5,
+      backoff: exponential,
+      initialDelay: 60000
+    },
+
+    // Not found - abort
+    NotFoundError -> abort "Repository not found",
+
+    // Fallback - log error and skip
+    _ -> skip
+  }
+}