reqon-dsl 0.2.0

package/docusaurus/docs/intro.md

@@ -0,0 +1,90 @@
+---
+sidebar_position: 1
+slug: /
+---
+
+# Introduction to Reqon
+
+Reqon is a **declarative DSL framework for building fetch, map, validate pipelines**. It's designed to make API data synchronization simple, reliable, and maintainable.
+
+## What is Reqon?
+
+Reqon allows you to define data pipelines using a clean, readable syntax. Instead of writing imperative code to fetch data, transform it, validate it, and store it, you declare *what* you want to happen, and Reqon handles *how* it happens.
+
+```vague
+mission SyncCustomers {
+  source API { auth: bearer, base: "https://api.example.com" }
+  store customers: file("customers")
+
+  action FetchCustomers {
+    get "/customers" { paginate: offset(page, 100) }
+    for customer in response.data {
+      map customer -> Customer {
+        id: .id,
+        name: .name,
+        email: .email
+      }
+      store customer -> customers { key: .id }
+    }
+  }
+
+  run FetchCustomers
+}
+```
+
+## Why Reqon?
+
+### Declarative by Design
+
+Traditional data pipelines require writing boilerplate code for HTTP requests, pagination, retries, error handling, and more. Reqon abstracts these concerns into a clean DSL, letting you focus on your business logic.
+
+### Built-in Best Practices
+
+- **Automatic pagination** - Handle offset, page number, or cursor-based pagination
+- **Retry with backoff** - Exponential, linear, or constant backoff strategies
+- **Rate limiting** - Respects API rate limits automatically
+- **Circuit breaker** - Prevents cascading failures
+- **Incremental sync** - Only fetch changes since last run
+
+### Extensible Architecture
+
+Reqon is built on [Vague](https://github.com/mcclowes/vague), a general-purpose expression DSL. This means you get powerful expression syntax for transformations and validations, while Reqon adds the execution semantics for data pipelines.
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────┐
+│                  Reqon                      │
+│  ┌─────────────────────────────────────┐    │
+│  │  Missions, Actions, Pipelines       │    │
+│  │  HTTP, Stores, Scheduling           │    │
+│  └─────────────────────────────────────┘    │
+│                    │                        │
+│  ┌─────────────────▼─────────────────┐      │
+│  │              Vague                 │     │
+│  │  Lexer, Parser, Expressions        │     │
+│  │  Match, Schema, Types              │     │
+│  └───────────────────────────────────┘      │
+└─────────────────────────────────────────────┘
+```
+
+- **Vague** provides the DSL layer (lexer, parser, expression syntax)
+- **Reqon** provides the runtime/framework with execution semantics
+
+For expression syntax, pattern matching, and schema definitions, refer to the [Vague documentation](https://github.com/mcclowes/vague).
+
+## Key Concepts
+
+| Concept | Description |
+|---------|-------------|
+| **Mission** | A complete data pipeline with sources, stores, and actions |
+| **Source** | An API endpoint with authentication configuration |
+| **Store** | A storage backend (memory, file, SQL, NoSQL) |
+| **Action** | A sequence of steps that process data |
+| **Step** | A single operation (fetch, map, validate, store, etc.) |
+
+## Next Steps
+
+- [Getting Started](./getting-started) - Install Reqon and run your first mission
+- [Core Concepts](./category/core-concepts) - Learn about missions, actions, and more
+- [Examples](./examples) - See real-world examples

package/docusaurus/docs/openapi/loading-specs.md

@@ -0,0 +1,305 @@
+---
+sidebar_position: 2
+---
+
+# Loading OpenAPI Specs
+
+Reqon can load OpenAPI specifications from files or URLs.
+
+## Local Files
+
+### YAML Format
+
+```vague
+source API from "./openapi.yaml" {
+  auth: bearer
+}
+```
+
+### JSON Format
+
+```vague
+source API from "./openapi.json" {
+  auth: bearer
+}
+```
+
+### Relative Paths
+
+```vague
+// Relative to mission file
+source API from "./specs/api.yaml"
+
+// Absolute path
+source API from "/home/user/specs/api.yaml"
+```
+
+## Remote URLs
+
+### Public Specs
+
+```vague
+source Petstore from "https://petstore3.swagger.io/api/v3/openapi.json" {
+  auth: none
+}
+```
+
+### Authenticated Specs
+
+```vague
+source PrivateAPI from "https://api.company.com/openapi.json" {
+  auth: bearer,
+  specAuth: {
+    type: "bearer",
+    token: env("SPEC_TOKEN")
+  }
+}
+```
+
+## Spec Caching
+
+Reqon caches resolved specs:
+
+```vague
+// First run: downloads and caches
+source API from "https://api.example.com/openapi.json"
+
+// Subsequent runs: uses cache
+```
+
+### Cache Location
+
+```
+.vague-data/
+└── oas-cache/
+    └── api.example.com-openapi.json
+```
+
+### Force Refresh
+
+```bash
+# Clear cache
+rm -rf .vague-data/oas-cache/
+
+# Or use --no-cache flag
+reqon mission.vague --no-oas-cache
+```
+
+## Spec Structure
+
+### Minimum Required
+
+```yaml
+openapi: 3.0.0
+info:
+  title: My API
+  version: 1.0.0
+servers:
+  - url: https://api.example.com
+paths:
+  /items:
+    get:
+      operationId: listItems
+      responses:
+        '200':
+          description: Success
+```
+
+### Full Example
+
+```yaml
+openapi: 3.0.0
+info:
+  title: E-commerce API
+  version: 2.0.0
+  description: API for managing products and orders
+
+servers:
+  - url: https://api.example.com/v2
+    description: Production
+  - url: https://staging.api.example.com/v2
+    description: Staging
+
+paths:
+  /products:
+    get:
+      operationId: listProducts
+      summary: List all products
+      parameters:
+        - name: limit
+          in: query
+          schema:
+            type: integer
+            default: 20
+        - name: offset
+          in: query
+          schema:
+            type: integer
+            default: 0
+      responses:
+        '200':
+          description: Product list
+          content:
+            application/json:
+              schema:
+                type: array
+                items:
+                  $ref: '#/components/schemas/Product'
+
+  /products/{id}:
+    get:
+      operationId: getProduct
+      parameters:
+        - name: id
+          in: path
+          required: true
+          schema:
+            type: string
+      responses:
+        '200':
+          description: Product details
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/Product'
+
+components:
+  schemas:
+    Product:
+      type: object
+      properties:
+        id:
+          type: string
+        name:
+          type: string
+        price:
+          type: number
+      required:
+        - id
+        - name
+```
+
+## Server Selection
+
+### Default Server
+
+Uses first server in spec:
+
+```yaml
+servers:
+  - url: https://api.example.com  # Used by default
+  - url: https://staging.example.com
+```
+
+### Override Base URL
+
+```vague
+source API from "./spec.yaml" {
+  auth: bearer,
+  base: "https://custom.example.com"  # Overrides spec
+}
+```
+
+### Environment-Based
+
+```vague
+source API from "./spec.yaml" {
+  auth: bearer,
+  base: env("API_BASE_URL")
+}
+```
+
+## Reference Resolution
+
+Reqon resolves `$ref` references:
+
+```yaml
+# References within same file
+$ref: '#/components/schemas/Product'
+
+# External file references
+$ref: './schemas/product.yaml'
+```
+
+## Validation
+
+### On Load
+
+Reqon validates the spec structure:
+
+```
+[Reqon] Loading OpenAPI spec from ./api.yaml
+[Reqon] Spec valid: 15 operations found
+```
+
+### Invalid Specs
+
+```
+[Reqon] Error: Invalid OpenAPI spec
+  - Missing required field: info.version
+  - Invalid path: /items must start with /
+```
+
+## Multiple Specs
+
+### Per-Source
+
+```vague
+mission MultiAPI {
+  source Products from "./products.yaml" { auth: bearer }
+  source Orders from "./orders.yaml" { auth: bearer }
+  source Users from "./users.yaml" { auth: oauth2 }
+
+  action SyncAll {
+    call Products.listProducts
+    call Orders.listOrders
+    call Users.listUsers
+  }
+}
+```
+
+### Shared Components
+
+If specs share schemas, use a single bundled spec:
+
+```vague
+source API from "./bundled-api.yaml" { auth: bearer }
+```
+
+## Troubleshooting
+
+### "Spec not found"
+
+Check file path:
+
+```bash
+ls -la ./openapi.yaml
+```
+
+### "Invalid spec"
+
+Validate externally:
+
+```bash
+npx swagger-cli validate ./openapi.yaml
+```
+
+### "Operation not found"
+
+Check operation ID matches exactly:
+
+```yaml
+paths:
+  /items:
+    get:
+      operationId: listItems  # Must match call API.listItems
+```
+
+### Network Issues
+
+For remote specs:
+
+```bash
+# Test connectivity
+curl -I https://api.example.com/openapi.json
+```

package/docusaurus/docs/openapi/operation-calls.md

@@ -0,0 +1,314 @@
+---
+sidebar_position: 3
+---
+
+# Operation Calls
+
+Call OpenAPI operations using the `call` syntax with operation IDs.
+
+## Basic Syntax
+
+```vague
+call SourceName.operationId
+call SourceName.operationId { options }
+```
+
+## Simple Calls
+
+### GET Operations
+
+```vague
+// OpenAPI: GET /pets with operationId: listPets
+call Petstore.listPets
+
+// OpenAPI: GET /pets/{petId} with operationId: getPetById
+call Petstore.getPetById { params: { petId: "123" } }
+```
+
+### POST Operations
+
+```vague
+// OpenAPI: POST /pets with operationId: addPet
+call Petstore.addPet {
+  body: {
+    name: "Fluffy",
+    tag: "cat"
+  }
+}
+```
+
+### PUT/PATCH Operations
+
+```vague
+// PUT - full replacement
+call API.updateItem {
+  params: { id: "123" },
+  body: { name: "New Name", status: "active" }
+}
+
+// PATCH - partial update
+call API.patchItem {
+  params: { id: "123" },
+  body: { status: "inactive" }
+}
+```
+
+### DELETE Operations
+
+```vague
+call API.deleteItem {
+  params: { id: "123" }
+}
+```
+
+## Parameters
+
+### Path Parameters
+
+For `/pets/{petId}`:
+
+```vague
+call Petstore.getPetById {
+  params: { petId: "123" }
+}
+// Generates: GET /pets/123
+```
+
+### Query Parameters
+
+For `/pets?limit=10&status=available`:
+
+```vague
+call Petstore.listPets {
+  params: {
+    limit: 10,
+    status: "available"
+  }
+}
+```
+
+### Combined Parameters
+
+```vague
+call API.listUserOrders {
+  params: {
+    userId: "123",      // Path: /users/{userId}/orders
+    status: "pending",  // Query: ?status=pending
+    limit: 50           // Query: ?limit=50
+  }
+}
+// Generates: GET /users/123/orders?status=pending&limit=50
+```
+
+## Request Body
+
+### Simple Body
+
+```vague
+call API.createItem {
+  body: {
+    name: "Test Item",
+    price: 29.99
+  }
+}
+```
+
+### Dynamic Body
+
+```vague
+for item in items {
+  call API.createItem {
+    body: {
+      name: item.name,
+      price: item.price,
+      metadata: {
+        source: "sync",
+        timestamp: now()
+      }
+    }
+  }
+}
+```
+
+### From Variable
+
+```vague
+map data -> Payload {
+  name: .name,
+  status: "active"
+}
+
+call API.createItem { body: data }
+```
+
+## Headers
+
+### Custom Headers
+
+```vague
+call API.listItems {
+  headers: {
+    "X-Request-ID": uuid(),
+    "Accept-Language": "en-US"
+  }
+}
+```
+
+### Tenant Headers
+
+```vague
+call Xero.listInvoices {
+  headers: {
+    "Xero-Tenant-Id": env("XERO_TENANT_ID")
+  }
+}
+```
+
+## Pagination with Operations
+
+```vague
+call API.listItems {
+  params: { limit: 100 },
+  paginate: offset(offset, 100),
+  until: length(response.items) == 0
+}
+```
+
+### Cursor Pagination
+
+```vague
+call API.listItems {
+  paginate: cursor(cursor, 100, "meta.nextCursor"),
+  until: response.meta.nextCursor == null
+}
+```
+
+## Combining Options
+
+```vague
+call API.searchItems {
+  params: {
+    query: "test",
+    limit: 50
+  },
+  headers: {
+    "X-Custom": "value"
+  },
+  paginate: cursor(after, 50, "pageInfo.endCursor"),
+  until: response.pageInfo.hasNextPage == false,
+  retry: {
+    maxAttempts: 3,
+    backoff: exponential
+  }
+}
+```
+
+## Response Handling
+
+```vague
+action FetchWithHandling {
+  call API.getItem { params: { id: itemId } }
+
+  match response {
+    { data: item } -> store item -> items { key: .id },
+    { error: e } -> abort e,
+    _ -> abort "Unexpected response"
+  }
+}
+```
+
+## Operation Chaining
+
+```vague
+action CreateAndFetch {
+  // Create
+  call API.createItem {
+    body: { name: "New Item" }
+  }
+
+  // response.id from creation
+  call API.getItem {
+    params: { id: response.id }
+  }
+
+  store response -> items { key: .id }
+}
+```
+
+## Error Handling
+
+```vague
+call API.riskyOperation { params: { id: "123" } }
+
+match response {
+  { code: 400 } -> abort "Invalid request",
+  { code: 401 } -> jump RefreshToken then retry,
+  { code: 404 } -> skip,
+  { code: 429 } -> retry { delay: 60000 },
+  { code: 500 } -> retry { maxAttempts: 3 },
+  _ -> continue
+}
+```
+
+## Dynamic Operation Calls
+
+### Based on Condition
+
+```vague
+action SmartSync {
+  call API.checkItem { params: { id: item.id } }
+
+  match response {
+    { exists: false } -> {
+      call API.createItem { body: item }
+    },
+    { exists: true, needsUpdate: true } -> {
+      call API.updateItem { params: { id: item.id }, body: item }
+    },
+    _ -> continue
+  }
+}
+```
+
+## Best Practices
+
+### Match Operation IDs
+
+```yaml
+# In OpenAPI spec
+operationId: listUsers  # camelCase recommended
+```
+
+```vague
+call API.listUsers  # Match exactly
+```
+
+### Use Descriptive Operations
+
+```yaml
+# Good
+operationId: createInvoice
+operationId: getInvoiceById
+operationId: listInvoicesByCustomer
+
+# Avoid
+operationId: post1
+operationId: get2
+```
+
+### Handle All Response Codes
+
+```vague
+call API.operation
+
+match response {
+  { code: 200 } -> continue,
+  { code: 201 } -> continue,
+  { code: 204 } -> continue,
+  { code: 400 } -> abort "Bad request",
+  { code: 401 } -> jump RefreshAuth then retry,
+  { code: 404 } -> skip,
+  { code: 500 } -> retry { maxAttempts: 3 },
+  _ -> abort "Unexpected response"
+}
+```