reqon-dsl 0.2.0

package/docusaurus/docs/core-concepts/actions.md

@@ -0,0 +1,286 @@
+---
+sidebar_position: 2
+---
+
+# Actions
+
+An **Action** is a named sequence of steps that process data. Actions are the building blocks of your pipeline logic.
+
+## Basic Structure
+
+```vague
+action ActionName {
+  // Step 1
+  get "/endpoint"
+
+  // Step 2
+  for item in response {
+    // Nested steps
+  }
+
+  // Step 3
+  store response -> storeName { key: .id }
+}
+```
+
+## Step Types
+
+Actions can contain the following step types:
+
+| Step | Description |
+|------|-------------|
+| `get`, `post`, `put`, `patch`, `delete` | HTTP requests |
+| `call` | OAS operation call |
+| `for...in...where` | Iteration with optional filtering |
+| `map...->` | Data transformation |
+| `validate` | Constraint checking |
+| `store...->` | Data persistence |
+| `match` | Pattern matching with flow control |
+
+## HTTP Request Steps
+
+Fetch data from APIs:
+
+```vague
+action FetchData {
+  // Simple GET
+  get "/users"
+
+  // With query parameters
+  get "/users" { params: { limit: 100, offset: 0 } }
+
+  // With pagination
+  get "/users" {
+    paginate: offset(page, 100),
+    until: length(response.users) == 0
+  }
+
+  // POST with body
+  post "/users" {
+    body: {
+      name: "John",
+      email: "john@example.com"
+    }
+  }
+}
+```
+
+## Iteration Steps
+
+Process collections:
+
+```vague
+action ProcessUsers {
+  get "/users"
+
+  // Iterate all items
+  for user in response.users {
+    // Process each user
+  }
+
+  // With filtering
+  for user in response.users where .status == "active" {
+    // Process only active users
+  }
+}
+```
+
+## Transformation Steps
+
+Transform data shapes:
+
+```vague
+action TransformData {
+  get "/users"
+
+  for user in response.users {
+    map user -> StandardUser {
+      id: .id,
+      fullName: concat(.firstName, " ", .lastName),
+      email: lowercase(.email),
+      createdAt: parseDate(.created_at)
+    }
+
+    store user -> users { key: .id }
+  }
+}
+```
+
+See the [Vague documentation](https://github.com/mcclowes/vague) for expression syntax.
+
+## Validation Steps
+
+Check data constraints:
+
+```vague
+action ValidateData {
+  get "/users"
+
+  for user in response.users {
+    validate user {
+      assume .id is string,
+      assume length(.name) > 0,
+      assume .email contains "@",
+      assume .age >= 18
+    }
+
+    store user -> users { key: .id }
+  }
+}
+```
+
+## Store Steps
+
+Persist data:
+
+```vague
+action SaveData {
+  get "/users"
+
+  // Store entire response
+  store response -> allData
+
+  // Store with key
+  store response.users -> users { key: .id }
+
+  // Upsert mode
+  store response.users -> users { key: .id, upsert: true }
+
+  // Partial update
+  store response.users -> users { key: .id, partial: true }
+}
+```
+
+## Pattern Matching Steps
+
+Route data based on shape:
+
+```vague
+action HandleResponse {
+  get "/users"
+
+  match response {
+    { error: e, code: 401 } -> jump RefreshToken then retry,
+    { error: e, code: 429 } -> retry { maxAttempts: 5 },
+    { error: e } -> abort "API error",
+    { users: _ } -> continue,
+    _ -> abort "Unexpected response"
+  }
+}
+```
+
+## Nested Actions
+
+Actions can reference other actions via `jump`:
+
+```vague
+action Main {
+  get "/data"
+
+  match response {
+    AuthError -> jump RefreshAuth then retry,
+    _ -> continue
+  }
+}
+
+action RefreshAuth {
+  post "/auth/refresh" { body: { token: env("REFRESH_TOKEN") } }
+  // Token is automatically used for subsequent requests
+}
+```
+
+## Action Composition in Pipelines
+
+Actions are composed in the `run` statement:
+
+```vague
+mission DataPipeline {
+  action Fetch { /* ... */ }
+  action Transform { /* ... */ }
+  action Export { /* ... */ }
+
+  // Sequential
+  run Fetch then Transform then Export
+
+  // Parallel groups
+  run [FetchA, FetchB] then Merge then Export
+}
+```
+
+## Variable Scope
+
+Variables are scoped to their action and nested contexts:
+
+```vague
+action ProcessData {
+  get "/users"  // response is set
+
+  for user in response.users {
+    // user is available here
+    // response is still available
+
+    map user -> processed {
+      // user and response available
+    }
+    // processed is available
+  }
+
+  // user is no longer available here
+  // response is still available
+}
+```
+
+## Best Practices
+
+### Single Responsibility
+
+Each action should do one thing well:
+
+```vague
+// Good: focused actions
+action FetchUsers {
+  get "/users"
+  store response -> rawUsers
+}
+
+action TransformUsers {
+  for user in rawUsers {
+    map user -> StandardUser { /* ... */ }
+    store user -> users { key: .id }
+  }
+}
+
+// Avoid: doing too much
+action DoEverything {
+  get "/users"
+  get "/orders"
+  // transform both
+  // export to multiple places
+}
+```
+
+### Handle Errors at Action Boundaries
+
+```vague
+action FetchWithErrorHandling {
+  get "/users"
+
+  match response {
+    ErrorResponse -> queue failures { item: response },
+    _ -> store response -> users { key: .id }
+  }
+}
+```
+
+### Use Descriptive Names
+
+```vague
+// Good
+action FetchActiveCustomersWithOrders { }
+action TransformToQuickBooksFormat { }
+action ExportToDataWarehouse { }
+
+// Avoid
+action Step1 { }
+action Process { }
+```

package/docusaurus/docs/core-concepts/missions.md

@@ -0,0 +1,264 @@
+---
+sidebar_position: 1
+---
+
+# Missions
+
+A **Mission** is the top-level container in Reqon. It defines a complete data pipeline, including all the sources, stores, schemas, actions, and the execution flow.
+
+## Basic Structure
+
+```vague
+mission MissionName {
+  // Source definitions (APIs)
+  source SourceName { auth: type, base: "url" }
+
+  // Store definitions (data persistence)
+  store storeName: adapter("identifier")
+
+  // Schema definitions (for validation and matching)
+  schema SchemaName { field: type }
+
+  // Action definitions (processing logic)
+  action ActionName {
+    // steps...
+  }
+
+  // Pipeline definition (execution order)
+  run ActionName then AnotherAction
+}
+```
+
+## Mission Components
+
+### Sources
+
+Define the APIs your mission connects to:
+
+```vague
+mission DataSync {
+  source MainAPI {
+    auth: bearer,
+    base: "https://api.example.com/v1"
+  }
+
+  source BackupAPI {
+    auth: api_key,
+    base: "https://backup.example.com"
+  }
+}
+```
+
+### Stores
+
+Define where data is persisted:
+
+```vague
+mission DataSync {
+  store customers: file("customers")
+  store orders: memory("orders")
+  store products: sql("products_table")
+}
+```
+
+### Schemas
+
+Define data shapes for validation and pattern matching:
+
+```vague
+mission DataSync {
+  schema Customer {
+    id: string,
+    name: string,
+    email: string,
+    createdAt: date?
+  }
+
+  schema ErrorResponse {
+    error: string,
+    code: number
+  }
+}
+```
+
+For schema syntax details, see the [Vague documentation](https://github.com/mcclowes/vague).
+
+### Actions
+
+Define the processing logic:
+
+```vague
+mission DataSync {
+  action FetchCustomers {
+    get "/customers"
+    store response -> customers { key: .id }
+  }
+
+  action ProcessOrders {
+    for customer in customers {
+      get "/orders" { params: { customerId: customer.id } }
+      store response -> orders { key: .id }
+    }
+  }
+}
+```
+
+### Pipeline
+
+Define execution order:
+
+```vague
+mission DataSync {
+  // Sequential execution
+  run FetchCustomers then ProcessOrders
+
+  // Or parallel execution
+  run [FetchProducts, FetchCategories] then MergeData
+}
+```
+
+## Multiple Missions
+
+A Reqon file can contain multiple missions:
+
+```vague
+mission SyncCustomers {
+  source API { auth: bearer, base: "https://api.example.com" }
+  store customers: file("customers")
+
+  action Fetch {
+    get "/customers"
+    store response -> customers { key: .id }
+  }
+
+  run Fetch
+}
+
+mission SyncOrders {
+  source API { auth: bearer, base: "https://api.example.com" }
+  store orders: file("orders")
+
+  action Fetch {
+    get "/orders"
+    store response -> orders { key: .id }
+  }
+
+  run Fetch
+}
+```
+
+## Scheduled Missions
+
+Add a schedule to run missions automatically:
+
+```vague
+mission DailySync {
+  schedule: every 6 hours
+
+  source API { auth: bearer, base: "https://api.example.com" }
+  store data: file("data")
+
+  action Sync {
+    get "/data" { since: lastSync }
+    store response -> data { key: .id }
+  }
+
+  run Sync
+}
+```
+
+See [Scheduling](../category/scheduling) for more details.
+
+## Mission Options
+
+Missions can include additional options:
+
+```vague
+mission RobustSync {
+  // Scheduling
+  schedule: cron "0 */6 * * *"
+
+  // Concurrency control
+  maxConcurrency: 5
+
+  // Skip if already running
+  skipIfRunning: true
+
+  // Retry on failure
+  retryOnFailure: {
+    maxAttempts: 3,
+    backoff: exponential
+  }
+
+  // ... sources, stores, actions ...
+}
+```
+
+## Best Practices
+
+### Keep Missions Focused
+
+Each mission should have a single responsibility:
+
+```vague
+// Good: focused mission
+mission SyncInvoices {
+  // Only deals with invoices
+}
+
+mission SyncPayments {
+  // Only deals with payments
+}
+```
+
+### Use Descriptive Names
+
+```vague
+// Good
+mission SyncXeroInvoicesToQuickBooks { }
+
+// Avoid
+mission Sync1 { }
+```
+
+### Organize Complex Missions
+
+For complex pipelines, use [multi-file missions](../advanced/multi-file-missions):
+
+```
+missions/
+└── invoice-sync/
+    ├── mission.vague     # Main definition
+    ├── fetch.vague       # Fetch actions
+    ├── transform.vague   # Transform actions
+    └── export.vague      # Export actions
+```
+
+### Handle Errors Gracefully
+
+Always include error handling:
+
+```vague
+mission RobustSync {
+  action FetchData {
+    get "/data"
+
+    match response {
+      ErrorResponse -> queue dlq,
+      _ -> store response -> data { key: .id }
+    }
+  }
+}
+```
+
+## Execution Context
+
+When a mission runs, Reqon creates an execution context that includes:
+
+- **stores**: Map of named store adapters
+- **sources**: Map of named HTTP clients
+- **schemas**: Map of named schema definitions
+- **variables**: Runtime variables (loop variables, etc.)
+- **response**: Last HTTP response
+
+This context is passed through the entire pipeline, allowing actions to share data.