reqon-dsl 0.2.0 → 0.4.0

package/examples/README.md

@@ -1,112 +0,0 @@
-# Reqon Examples
-
-This directory contains examples demonstrating Reqon's features for declarative data pipelines.
-
-## Examples Overview
-
-| Example | Description | Key Features |
-|---------|-------------|--------------|
-| [jsonplaceholder](./jsonplaceholder/) | Basic public API sync | `auth: none`, fetch, map, for loops |
-| [petstore](./petstore/) | OpenAPI spec integration | OAS operationId, cursor pagination |
-| [xero](./xero/) | OAuth2 invoice sync | OAuth2, hydration, **match steps**, **flow control** |
-| [github-sync](./github-sync/) | Multi-file mission | **Folder structure**, **parallel execution**, schema matching |
-| [error-handling](./error-handling/) | Comprehensive error handling | **All flow control directives**, dead letter queues |
-| [temporal-comparison](./temporal-comparison/) | E-commerce reconciliation | Multi-source, **parallel execution**, rate limiting |
-
-## Feature Index
-
-### Multi-File Missions
-Organize large missions into folders with separate action files:
-```
-github-sync/
-├── mission.vague       # Sources, stores, schemas, pipeline
-├── fetch-issues.vague  # Action file
-├── fetch-prs.vague     # Action file
-└── normalize.vague     # Action file
-```
-See: [github-sync](./github-sync/)
-
-### Parallel Execution
-Run multiple actions concurrently:
-```vague
-run [FetchOrders, FetchPayments, FetchShipments] then Reconcile
-```
-See: [github-sync](./github-sync/), [temporal-comparison](./temporal-comparison/)
-
-### Schema Overloading with Match Steps
-Handle different API response types declaratively:
-```vague
-match response {
-  SuccessSchema -> { store response -> cache },
-  RateLimitError -> retry { maxAttempts: 5 },
-  AuthError -> jump RefreshToken then retry,
-  _ -> abort "Unexpected response"
-}
-```
-See: [xero](./xero/), [error-handling](./error-handling/)
-
-### Flow Control Directives
-Six directives for controlling execution flow:
-
-| Directive | Description | Example |
-|-----------|-------------|---------|
-| `continue` | Proceed to next step | `Schema -> continue` |
-| `skip` | Skip remaining steps | `Schema -> skip` |
-| `abort` | Halt mission | `Schema -> abort "Error"` |
-| `retry` | Retry with backoff | `Schema -> retry { maxAttempts: 5 }` |
-| `queue` | Send to dead letter queue | `Schema -> queue dlq` |
-| `jump` | Execute action, then continue | `Schema -> jump Refresh then retry` |
-
-See: [error-handling](./error-handling/)
-
-### Authentication Types
-```vague
-source API { auth: none }           # Public API
-source API { auth: bearer }         # Bearer token
-source API { auth: oauth2 }         # OAuth2
-source API { auth: basic }          # Basic auth
-source API { auth: api_key }        # API key
-```
-
-### Pagination Strategies
-```vague
-paginate: offset(page, 100)                    # Offset pagination
-paginate: page(page, 100)                      # Page number pagination
-paginate: cursor(cursor, 100, "nextCursor")   # Cursor pagination
-```
-
-## Running Examples
-
-```bash
-# Build first
-npm run build
-
-# Run any example
-node dist/cli.js examples/<example>/<file>.vague --verbose
-
-# Run multi-file mission (folder)
-node dist/cli.js examples/github-sync --verbose
-
-# Dry run (no actual API calls)
-node dist/cli.js examples/xero/invoices.vague --dry-run
-
-# With credentials
-node dist/cli.js examples/xero/invoices.vague --auth credentials.json
-```
-
-## Credentials Format
-
-Create a `credentials.json` file:
-```json
-{
-  "SourceName": {
-    "type": "bearer",
-    "token": "your-token"
-  },
-  "OAuthSource": {
-    "type": "oauth2",
-    "accessToken": "your-access-token",
-    "refreshToken": "your-refresh-token"
-  }
-}
-```

package/examples/error-handling/README.md

@@ -1,150 +0,0 @@
-# Error Handling Example
-
-Demonstrates comprehensive error handling with schema matching and all flow control directives.
-
-## What it does
-
-1. **FetchPendingPayments**: Fetches payments with retry and auth refresh
-2. **CheckFraudRisk**: Checks each payment for fraud, routing high-risk to review queue
-3. **ProcessPayments**: Captures approved payments with full error handling
-4. **GenerateReport**: Creates a summary of processing results
-
-## Run
-
-```bash
-node dist/cli.js examples/error-handling/payment-processor.vague --auth credentials.json --verbose
-```
-
-## Flow Control Directives
-
-This example demonstrates all six flow control directives:
-
-### 1. `continue` - Proceed to next step
-```vague
-match response {
-  FraudWarning where .risk_level == "low" -> continue
-}
-```
-When the condition matches, continue with the next steps in the action.
-
-### 2. `skip` - Skip remaining steps in current iteration
-```vague
-match response {
-  _ -> skip  // Unknown response - skip this payment, move to next
-}
-```
-Skips remaining steps for the current item in a `for` loop but continues with the next item.
-
-### 3. `abort "message"` - Halt mission with error
-```vague
-match response {
-  ServerError -> abort "Payment gateway unavailable"
-}
-```
-Immediately stops the entire mission and reports the error.
-
-### 4. `retry { config }` - Retry with backoff
-```vague
-match response {
-  RateLimitError -> retry {
-    maxAttempts: 5,
-    backoff: exponential,    // or: linear, constant
-    initialDelay: 5000,
-    maxDelay: 120000
-  }
-}
-```
-Retries the current fetch with configurable backoff strategy.
-
-### 5. `queue target` - Send to dead-letter queue
-```vague
-match response {
-  ServerError -> queue dead_letter_queue
-}
-```
-Parks the current item for later processing or manual review.
-
-### 6. `jump Action then retry` - Execute action, then continue
-```vague
-match response {
-  AuthenticationError -> jump RefreshAuth then retry
-}
-```
-Jumps to another action (e.g., to refresh auth), then retries the original request.
-
-## Schema Matching Patterns
-
-### Simple Schema Match
-```vague
-match response {
-  PaymentSuccess -> { store response -> payments }
-}
-```
-
-### Schema with Guard Clause
-```vague
-match response {
-  FraudWarning where .risk_level == "high" -> queue fraud_review
-}
-```
-
-### Array Schema Match
-```vague
-match response {
-  [PaymentSuccess] -> { store response -> payments { key: .id } }
-}
-```
-
-### Wildcard (Catch-All)
-```vague
-match response {
-  SuccessSchema -> continue,
-  ErrorSchema -> abort "Error occurred",
-  _ -> skip  // Handle any other response
-}
-```
-
-## Error Handling Patterns
-
-### 1. Retry with Auth Refresh
-```vague
-AuthenticationError -> jump RefreshAuth then retry
-```
-
-### 2. Progressive Backoff
-```vague
-RateLimitError -> retry {
-  maxAttempts: 5,
-  backoff: exponential,
-  initialDelay: 1000,
-  maxDelay: 60000
-}
-```
-
-### 3. Dead Letter Queue for Failures
-```vague
-ServerError -> queue dead_letter_queue
-```
-
-### 4. Conditional Processing
-```vague
-match response {
-  FraudWarning where .risk_level == "low" -> continue,
-  FraudWarning where .risk_level == "medium" -> {
-    store { ... } -> review_queue
-  },
-  FraudWarning where .risk_level == "high" -> {
-    store { ... } -> fraud_queue
-  }
-}
-```
-
-## Features demonstrated
-
-- All 6 flow control directives
-- Schema matching with guard clauses
-- Retry with exponential/linear/constant backoff
-- Token refresh via `jump...then retry`
-- Dead letter queue pattern
-- Conditional routing based on response content
-- Error logging and audit trails

package/examples/error-handling/payment-processor.vague

@@ -1,287 +0,0 @@
-// Payment Processing Pipeline with Comprehensive Error Handling
-// Demonstrates all flow control directives: continue, skip, abort, retry, queue, jump
-
-mission ProcessPayments {
-
-  // ============================================================
-  // SOURCES
-  // ============================================================
-
-  source PaymentGateway {
-    auth: bearer,
-    base: "https://api.paymentgateway.com/v1",
-    rateLimit: {
-      strategy: "pause",
-      maxWait: 60,
-      fallbackRpm: 100
-    }
-  }
-
-  source FraudDetection {
-    auth: api_key,
-    base: "https://fraud.example.com/api"
-  }
-
-  // ============================================================
-  // STORES
-  // ============================================================
-
-  store pending_payments: memory("pending")
-  store processed_payments: sql("payments")
-  store failed_payments: memory("failed")
-  store fraud_review_queue: memory("fraud_review")
-  store dead_letter_queue: memory("dlq")
-
-  // ============================================================
-  // SCHEMAS - Response Types
-  // ============================================================
-
-  schema PaymentSuccess {
-    id: string,
-    status: string,
-    amount: decimal,
-    currency: string,
-    captured_at: date
-  }
-
-  schema PaymentPending {
-    id: string,
-    status: string,
-    requires_action: boolean
-  }
-
-  schema RateLimitError {
-    error: string,
-    retry_after: int
-  }
-
-  schema AuthenticationError {
-    error: string,
-    code: string
-  }
-
-  schema ValidationError {
-    error: string,
-    field: string,
-    message: string
-  }
-
-  schema FraudWarning {
-    risk_score: decimal,
-    risk_level: string,
-    recommendation: string
-  }
-
-  schema ServerError {
-    error: string,
-    request_id: string
-  }
-
-  // ============================================================
-  // ACTION: Fetch Pending Payments
-  // ============================================================
-
-  action FetchPendingPayments {
-    get "/payments" {
-      body: { "status": "pending" },
-      retry: {
-        maxAttempts: 3,
-        backoff: "exponential",
-        initialDelay: 1000
-      }
-    }
-
-    match response {
-      // Success - store and continue
-      [PaymentSuccess] -> {
-        store response -> pending_payments { key: .id }
-      },
-
-      // Rate limited - retry with backoff
-      RateLimitError -> retry {
-        maxAttempts: 5,
-        backoff: exponential,
-        initialDelay: 5000,
-        maxDelay: 120000
-      },
-
-      // Auth error - refresh token and retry
-      AuthenticationError -> jump RefreshAuth then retry,
-
-      // Server error - abort the mission
-      ServerError -> abort "Payment gateway unavailable",
-
-      // Unexpected response - log and continue with empty
-      _ -> {
-        store {
-          error: "Unexpected response fetching payments",
-          response: response,
-          timestamp: now()
-        } -> failed_payments { key: now() }
-      }
-    }
-  }
-
-  // ============================================================
-  // ACTION: Refresh Authentication (called via jump)
-  // ============================================================
-
-  action RefreshAuth {
-    post "/auth/refresh" {
-      source: PaymentGateway,
-      body: {
-        "refresh_token": env("PAYMENT_REFRESH_TOKEN")
-      }
-    }
-
-    match response {
-      _ where .access_token != null -> continue,
-      _ -> abort "Failed to refresh authentication"
-    }
-  }
-
-  // ============================================================
-  // ACTION: Check Fraud Risk
-  // ============================================================
-
-  action CheckFraudRisk {
-    for payment in pending_payments {
-      post "/check" {
-        source: FraudDetection,
-        body: {
-          "payment_id": payment.id,
-          "amount": payment.amount,
-          "currency": payment.currency
-        }
-      }
-
-      match response {
-        // Low risk - continue processing
-        FraudWarning where .risk_level == "low" -> continue,
-
-        // Medium risk - queue for human review, skip this payment
-        FraudWarning where .risk_level == "medium" -> {
-          store {
-            payment_id: payment.id,
-            risk_score: response.risk_score,
-            reason: "Medium risk - requires review"
-          } -> fraud_review_queue { key: payment.id }
-        },
-
-        // High risk - queue and skip
-        FraudWarning where .risk_level == "high" -> {
-          store {
-            payment_id: payment.id,
-            risk_score: response.risk_score,
-            reason: "High fraud risk detected"
-          } -> fraud_review_queue { key: payment.id }
-        },
-
-        // Fraud service error - queue to DLQ for later processing
-        ServerError -> queue dead_letter_queue,
-
-        // Rate limited - retry
-        RateLimitError -> retry {
-          maxAttempts: 3,
-          backoff: constant,
-          initialDelay: 10000
-        },
-
-        // Unknown - skip but log
-        _ -> skip
-      }
-    }
-  }
-
-  // ============================================================
-  // ACTION: Process Approved Payments
-  // ============================================================
-
-  action ProcessPayments {
-    // Only process payments not in fraud review
-    for payment in pending_payments where not exists(fraud_review_queue[payment.id]) {
-      post "/payments/{payment.id}/capture" {
-        source: PaymentGateway,
-        body: {
-          "idempotency_key": payment.id + "_capture"
-        }
-      }
-
-      match response {
-        // Successfully captured
-        PaymentSuccess -> {
-          store response -> processed_payments {
-            key: .id,
-            upsert: true
-          }
-        },
-
-        // Payment requires additional action (3DS, etc)
-        PaymentPending where .requires_action == true -> {
-          store {
-            payment_id: payment.id,
-            reason: "Requires customer action",
-            status: "pending_action"
-          } -> failed_payments { key: payment.id }
-        },
-
-        // Validation error - payment data issue
-        ValidationError -> {
-          store {
-            payment_id: payment.id,
-            error: response.message,
-            field: response.field,
-            status: "invalid"
-          } -> failed_payments { key: payment.id }
-        },
-
-        // Rate limited
-        RateLimitError -> retry {
-          maxAttempts: 3,
-          backoff: exponential,
-          initialDelay: 2000
-        },
-
-        // Auth expired during processing
-        AuthenticationError -> jump RefreshAuth then retry,
-
-        // Server error - queue for retry later
-        ServerError -> queue dead_letter_queue,
-
-        // Catch-all - queue to DLQ
-        _ -> queue dead_letter_queue
-      }
-    }
-  }
-
-  // ============================================================
-  // ACTION: Generate Report
-  // ============================================================
-
-  action GenerateReport {
-    map {
-      processed: length(processed_payments),
-      failed: length(failed_payments),
-      fraud_flagged: length(fraud_review_queue),
-      queued_for_retry: length(dead_letter_queue),
-      completed_at: now()
-    } -> report
-
-    validate report {
-      assume .processed >= 0
-    }
-
-    store response -> processed_payments {
-      key: "report_" + now()
-    }
-  }
-
-  // ============================================================
-  // PIPELINE
-  // ============================================================
-
-  run FetchPendingPayments
-    then CheckFraudRisk
-    then ProcessPayments
-    then GenerateReport
-}

package/examples/github-sync/README.md

@@ -1,74 +0,0 @@
-# GitHub Sync Example
-
-Demonstrates multi-file missions, parallel execution, and error handling with flow control.
-
-## Folder Structure
-
-This mission is organized as a folder with separate action files:
-
-```
-github-sync/
-├── mission.vague       # Main file: sources, stores, schemas, pipeline
-├── fetch-issues.vague  # Fetches issues with pagination and error handling
-├── fetch-prs.vague     # Fetches pull requests with error handling
-├── normalize.vague     # Normalizes to unified schema
-└── README.md
-```
-
-## What it does
-
-1. **FetchIssues** and **FetchPRs** run in **parallel** to fetch data concurrently
-2. **Normalize** runs after both complete, unifying issues and PRs into `work_items`
-
-## Run
-
-```bash
-# Run from folder path
-node dist/cli.js examples/github-sync --auth credentials.json --verbose
-
-# Environment variables needed
-export GITHUB_OWNER=your-org
-export GITHUB_REPO=your-repo
-```
-
-Requires a `credentials.json`:
-```json
-{
-  "GitHub": {
-    "type": "bearer",
-    "token": "ghp_your_github_token"
-  }
-}
-```
-
-## Features demonstrated
-
-### Multi-file Missions
-- `mission.vague` contains sources, stores, schemas, and pipeline
-- Action files (`fetch-issues.vague`, etc.) are automatically discovered and merged
-- Actions can reference stores and schemas defined in the root file
-
-### Parallel Execution
-```vague
-run [FetchIssues, FetchPRs] then Normalize
-```
-- `[FetchIssues, FetchPRs]` runs both actions concurrently
-- `then Normalize` waits for both to complete before running
-
-### Schema Overloading with Match Steps
-```vague
-match response {
-  [GitHubIssue] -> { store response -> issues_raw },
-  RateLimitError -> retry { maxAttempts: 5 },
-  NotFoundError -> abort "Repository not found",
-  _ -> skip
-}
-```
-
-### Flow Control Directives
-- `continue` - Proceed to next step
-- `skip` - Skip remaining steps in current action
-- `abort "message"` - Halt mission with error
-- `retry { ... }` - Retry with backoff configuration
-- `queue target` - Send to dead-letter queue
-- `jump ActionName then retry` - Execute another action, then retry

package/examples/github-sync/fetch-issues.vague

@@ -1,47 +0,0 @@
-// Fetch GitHub Issues with error handling
-// External action file - automatically merged into mission
-
-action FetchIssues {
-  get "/repos/{GITHUB_OWNER}/{GITHUB_REPO}/issues" {
-    source: GitHub,
-    body: {
-      "state": "all",
-      "per_page": 100
-    },
-    paginate: page(page, 100),
-    until: length(response) == 0,
-    retry: {
-      maxAttempts: 3,
-      backoff: "exponential",
-      initialDelay: 1000
-    }
-  }
-
-  // Handle different response types with schema matching
-  match response {
-    // Success - continue processing
-    [GitHubIssue] -> {
-      store response -> issues_raw { key: .id }
-    },
-
-    // Rate limited - retry with backoff
-    RateLimitError -> retry {
-      maxAttempts: 5,
-      backoff: exponential,
-      initialDelay: 60000,
-      maxDelay: 300000
-    },
-
-    // Repo not found - abort mission
-    NotFoundError -> abort "Repository not found - check GITHUB_OWNER and GITHUB_REPO",
-
-    // Unexpected response - log and skip
-    _ -> {
-      store {
-        error: "Unexpected response in FetchIssues",
-        response: response,
-        timestamp: now()
-      } -> errors { key: now() }
-    }
-  }
-}

package/examples/github-sync/fetch-prs.vague

@@ -1,40 +0,0 @@
-// Fetch GitHub Pull Requests with error handling
-// External action file - automatically merged into mission
-
-action FetchPRs {
-  get "/repos/{GITHUB_OWNER}/{GITHUB_REPO}/pulls" {
-    source: GitHub,
-    body: {
-      "state": "all",
-      "per_page": 100
-    },
-    paginate: page(page, 100),
-    until: length(response) == 0,
-    retry: {
-      maxAttempts: 3,
-      backoff: "exponential",
-      initialDelay: 1000
-    }
-  }
-
-  // Handle different response types
-  match response {
-    // Success - store PRs
-    [GitHubPR] -> {
-      store response -> prs_raw { key: .id }
-    },
-
-    // Rate limited - retry
-    RateLimitError -> retry {
-      maxAttempts: 5,
-      backoff: exponential,
-      initialDelay: 60000
-    },
-
-    // Not found - abort
-    NotFoundError -> abort "Repository not found",
-
-    // Fallback - log error and skip
-    _ -> skip
-  }
-}