reqon-dsl 0.2.0 → 0.4.0

package/docusaurus/docs/getting-started.md

@@ -1,256 +0,0 @@
----
-sidebar_position: 2
----
-
-# Getting Started
-
-This guide will help you install Reqon and run your first data pipeline.
-
-## Prerequisites
-
-- Node.js 18 or later
-- npm or yarn
-
-## Installation
-
-```bash
-npm install reqon
-```
-
-Or if you prefer yarn:
-
-```bash
-yarn add reqon
-```
-
-## Your First Mission
-
-Create a file called `hello.vague`:
-
-```vague
-mission HelloWorld {
-  source JSONPlaceholder {
-    auth: none,
-    base: "https://jsonplaceholder.typicode.com"
-  }
-
-  store posts: memory("posts")
-
-  action FetchPosts {
-    get "/posts"
-
-    for post in response {
-      map post -> Post {
-        id: .id,
-        title: .title,
-        body: .body,
-        userId: .userId
-      }
-      store post -> posts { key: .id }
-    }
-  }
-
-  run FetchPosts
-}
-```
-
-## Running Your Mission
-
-### Using the CLI
-
-```bash
-npx reqon hello.vague
-```
-
-With verbose output:
-
-```bash
-npx reqon hello.vague --verbose
-```
-
-### Using the API
-
-```typescript
-import { execute } from 'reqon';
-
-const result = await execute(`
-  mission HelloWorld {
-    source JSONPlaceholder { auth: none, base: "https://jsonplaceholder.typicode.com" }
-    store posts: memory("posts")
-
-    action FetchPosts {
-      get "/posts"
-      store response -> posts { key: .id }
-    }
-
-    run FetchPosts
-  }
-`);
-
-console.log(`Fetched ${result.stores.get('posts')?.list().length} posts`);
-```
-
-## Understanding the Output
-
-When you run a mission, Reqon provides detailed execution information:
-
-```
-[Reqon] Starting mission: HelloWorld
-[Reqon] Running action: FetchPosts
-[Reqon] GET https://jsonplaceholder.typicode.com/posts
-[Reqon] Stored 100 items to posts
-[Reqon] Mission completed in 234ms
-```
-
-## Mission Structure
-
-Every Reqon mission follows this structure:
-
-```vague
-mission MissionName {
-  // 1. Define data sources (APIs)
-  source SourceName { auth: type, base: "url" }
-
-  // 2. Define storage targets
-  store storeName: adapter("name")
-
-  // 3. Define schemas (optional, for validation)
-  schema SchemaName { field: type }
-
-  // 4. Define actions (processing steps)
-  action ActionName {
-    // Steps: fetch, map, validate, store, for, match
-  }
-
-  // 5. Define the pipeline
-  run ActionName then AnotherAction
-}
-```
-
-## Adding Transformations
-
-Use `map` to transform data:
-
-```vague
-action TransformPosts {
-  get "/posts"
-
-  for post in response {
-    map post -> BlogPost {
-      id: .id,
-      title: .title,
-      excerpt: substring(.body, 0, 100),
-      author: concat("User ", toString(.userId))
-    }
-    store post -> posts { key: .id }
-  }
-}
-```
-
-## Adding Validation
-
-Use `validate` to check constraints:
-
-```vague
-action ValidatedFetch {
-  get "/posts"
-
-  for post in response {
-    validate post {
-      assume .id > 0,
-      assume length(.title) > 0,
-      assume .userId is number
-    }
-    store post -> posts { key: .id }
-  }
-}
-```
-
-## Handling Pagination
-
-Most APIs require pagination. Reqon makes this easy:
-
-```vague
-action FetchAllPosts {
-  get "/posts" {
-    paginate: offset(page, 20),
-    until: length(response) == 0
-  }
-
-  store response -> posts { key: .id }
-}
-```
-
-## Error Handling
-
-Use `match` for pattern-based error handling:
-
-```vague
-action RobustFetch {
-  get "/posts"
-
-  match response {
-    { error: _ } -> abort "API returned error",
-    { data: _ } -> continue,
-    _ -> store response -> posts { key: .id }
-  }
-}
-```
-
-## Next Steps
-
-Now that you've run your first mission, explore these topics:
-
-- [Core Concepts](./category/core-concepts) - Understand missions, actions, sources, and stores
-- [DSL Syntax](./category/dsl-syntax) - Learn the complete Reqon syntax
-- [Authentication](./category/authentication) - Connect to authenticated APIs
-- [Examples](./examples) - See more complex examples
-
-## Common Issues
-
-### "Source not found" Error
-
-Make sure you've defined the source before using it in an action:
-
-```vague
-mission Example {
-  source API { auth: none, base: "https://api.example.com" }  // Define first
-
-  action FetchData {
-    get "/data"  // Uses the default source
-  }
-
-  run FetchData
-}
-```
-
-### "Store not found" Error
-
-Ensure stores are defined at the mission level:
-
-```vague
-mission Example {
-  store myData: memory("data")  // Define at mission level
-
-  action SaveData {
-    get "/data"
-    store response -> myData { key: .id }  // Use in actions
-  }
-
-  run SaveData
-}
-```
-
-### Network Errors
-
-Add retry configuration for unreliable networks:
-
-```vague
-get "/data" {
-  retry: {
-    maxAttempts: 3,
-    backoff: exponential,
-    initialDelay: 1000
-  }
-}
-```

package/docusaurus/docs/http/circuit-breaker.md

@@ -1,401 +0,0 @@
----
-sidebar_position: 6
----
-
-# Circuit Breaker
-
-The circuit breaker pattern prevents cascading failures when an API is experiencing problems. Reqon includes a built-in circuit breaker for robust error handling.
-
-## How It Works
-
-```
-     ┌────────────────────────────────────────┐
-     │                                        │
-     ▼                                        │
-┌─────────┐  failures > threshold  ┌─────────┐ │
-│ CLOSED  │ ───────────────────────│  OPEN   │ │
-│ (normal)│                        │ (fail)  │ │
-└────┬────┘                        └────┬────┘ │
-     │                                  │      │
-     │        ┌───────────┐             │      │
-     │        │ HALF_OPEN │◄────────────┘      │
-     │        │  (test)   │   after timeout   │
-     │        └─────┬─────┘                   │
-     │              │                          │
-     │   successes > threshold                 │
-     └──────────────┴──────────────────────────┘
-```
-
-**States:**
-- **CLOSED**: Normal operation, requests pass through
-- **OPEN**: Circuit tripped, requests fail immediately
-- **HALF_OPEN**: Testing if service recovered
-
-## Configuration
-
-```vague
-source API {
-  auth: bearer,
-  base: "https://api.example.com",
-  circuitBreaker: {
-    failureThreshold: 5,
-    resetTimeout: 30000,
-    successThreshold: 2,
-    failureWindow: 60000
-  }
-}
-```
-
-## Options
-
-| Option | Description | Default |
-|--------|-------------|---------|
-| `failureThreshold` | Failures before opening circuit | 5 |
-| `resetTimeout` | Time before trying half-open (ms) | 30000 |
-| `successThreshold` | Successes needed to close circuit | 2 |
-| `failureWindow` | Window for counting failures (ms) | 60000 |
-
-## States Explained
-
-### Closed State
-
-Normal operation:
-
-```vague
-// All requests pass through normally
-get "/data"  // Success
-get "/data"  // Success
-get "/data"  // Failure (1)
-get "/data"  // Failure (2)
-get "/data"  // Success - counter resets
-```
-
-Failures are counted within the `failureWindow`. Successes reset the counter.
-
-### Open State
-
-When `failureThreshold` is reached:
-
-```vague
-// After 5 consecutive failures...
-get "/data"  // Immediately fails - circuit is OPEN
-get "/data"  // Immediately fails - no actual request made
-```
-
-All requests fail fast without calling the API.
-
-### Half-Open State
-
-After `resetTimeout`:
-
-```vague
-// After 30 seconds...
-get "/data"  // Actually sent - testing if API is back
-
-// If success:
-get "/data"  // Sent - need 2 successes total
-// After 2 successes, circuit CLOSES
-
-// If failure:
-// Circuit goes back to OPEN
-```
-
-## Error Handling with Circuit Breaker
-
-```vague
-action FetchWithCircuitBreaker {
-  get "/data"
-
-  match response {
-    { error: "circuit_open" } -> {
-      // Circuit is open - API is down
-      store { status: "api_down", timestamp: now() } -> statusLog
-      abort "API unavailable - circuit breaker open"
-    },
-    { error: e } -> {
-      // Other errors - may trip circuit
-      abort e
-    },
-    _ -> continue
-  }
-}
-```
-
-## Combining with Retry
-
-```vague
-source API {
-  auth: bearer,
-  base: "https://api.example.com",
-  circuitBreaker: {
-    failureThreshold: 5,
-    resetTimeout: 30000
-  }
-}
-
-action RobustFetch {
-  get "/data" {
-    retry: {
-      maxAttempts: 3,
-      backoff: exponential
-    }
-  }
-
-  match response {
-    { error: "circuit_open" } -> skip,  // Don't retry if circuit is open
-    { error: _ } -> abort "Request failed",
-    _ -> store response -> data { key: .id }
-  }
-}
-```
-
-Order of operations:
-1. Check if circuit is open → fail fast if yes
-2. Make request
-3. If fails, retry logic kicks in
-4. Each failure counts toward circuit breaker
-5. After max retries, may trip circuit
-
-## Combining with Rate Limiting
-
-```vague
-source API {
-  auth: bearer,
-  base: "https://api.example.com",
-  rateLimit: {
-    requestsPerMinute: 60,
-    strategy: "pause"
-  },
-  circuitBreaker: {
-    failureThreshold: 5,
-    resetTimeout: 30000
-  }
-}
-```
-
-Both work together:
-- Rate limiter controls request pacing
-- Circuit breaker handles API failures
-
-## Per-Source Circuit Breakers
-
-Each source has its own circuit breaker:
-
-```vague
-mission MultiSourceSync {
-  source ReliableAPI {
-    auth: bearer,
-    base: "https://reliable.api.com",
-    circuitBreaker: {
-      failureThreshold: 10  // More tolerant
-    }
-  }
-
-  source FlakyAPI {
-    auth: bearer,
-    base: "https://flaky.api.com",
-    circuitBreaker: {
-      failureThreshold: 3,  // Trip quickly
-      resetTimeout: 60000   // Wait longer before retry
-    }
-  }
-
-  action FetchBoth {
-    // Each source's circuit is independent
-    get ReliableAPI "/data"
-    get FlakyAPI "/data"
-  }
-}
-```
-
-## Monitoring Circuit State
-
-```vague
-action MonitoredFetch {
-  get "/data"
-
-  match response {
-    { error: "circuit_open" } -> {
-      store {
-        event: "circuit_open",
-        source: "API",
-        timestamp: now()
-      } -> circuitEvents
-      skip
-    },
-    { error: e } -> {
-      store {
-        event: "failure",
-        error: e,
-        timestamp: now()
-      } -> circuitEvents
-      abort e
-    },
-    _ -> {
-      store {
-        event: "success",
-        timestamp: now()
-      } -> circuitEvents
-      continue
-    }
-  }
-}
-```
-
-## Fallback Patterns
-
-### Fallback to Cache
-
-```vague
-action FetchWithFallback {
-  get "/data"
-
-  match response {
-    { error: "circuit_open" } -> {
-      // Use cached data
-      for item in cachedData {
-        store item -> data { key: .id }
-      }
-      continue
-    },
-    { data: items } -> {
-      // Update cache
-      for item in items {
-        store item -> cachedData { key: .id }
-        store item -> data { key: .id }
-      }
-    },
-    _ -> abort "Unexpected response"
-  }
-}
-```
-
-### Fallback to Secondary Source
-
-```vague
-mission FallbackSync {
-  source Primary { circuitBreaker: { failureThreshold: 3 } }
-  source Secondary { circuitBreaker: { failureThreshold: 5 } }
-
-  action FetchWithFallback {
-    get Primary "/data"
-
-    match response {
-      { error: "circuit_open" } -> {
-        // Primary is down, try secondary
-        get Secondary "/data"
-        store response -> data { key: .id }
-      },
-      { data: _ } -> store response.data -> data { key: .id },
-      _ -> abort "Both sources failed"
-    }
-  }
-}
-```
-
-## Best Practices
-
-### Configure Based on API Behavior
-
-```vague
-// For stable APIs
-circuitBreaker: {
-  failureThreshold: 10,
-  resetTimeout: 30000
-}
-
-// For flaky APIs
-circuitBreaker: {
-  failureThreshold: 3,
-  resetTimeout: 60000
-}
-
-// For critical APIs (fail fast)
-circuitBreaker: {
-  failureThreshold: 2,
-  resetTimeout: 10000
-}
-```
-
-### Use with Error Handling
-
-```vague
-action RobustFetch {
-  get "/data"
-
-  match response {
-    { error: "circuit_open" } -> {
-      // Log and handle gracefully
-      store { event: "circuit_open" } -> logs
-      skip
-    },
-    _ -> continue
-  }
-}
-```
-
-### Set Appropriate Timeouts
-
-```vague
-// For fast recovery APIs
-circuitBreaker: {
-  resetTimeout: 10000  // 10 seconds
-}
-
-// For slow recovery APIs
-circuitBreaker: {
-  resetTimeout: 300000  // 5 minutes
-}
-```
-
-### Consider Failure Window
-
-```vague
-// For burst-tolerant scenarios
-circuitBreaker: {
-  failureThreshold: 5,
-  failureWindow: 60000  // 1 minute window
-}
-
-// For strict scenarios
-circuitBreaker: {
-  failureThreshold: 3,
-  failureWindow: 10000  // 10 second window
-}
-```
-
-## Troubleshooting
-
-### Circuit Trips Too Often
-
-Increase threshold or window:
-
-```vague
-circuitBreaker: {
-  failureThreshold: 10,  // More tolerance
-  failureWindow: 120000  // Longer window
-}
-```
-
-### Circuit Stays Open Too Long
-
-Decrease reset timeout:
-
-```vague
-circuitBreaker: {
-  resetTimeout: 10000  // Try sooner
-}
-```
-
-### False Positives
-
-Ensure only real failures count:
-
-```vague
-match response {
-  { code: 404 } -> skip,  // Not a failure
-  { code: 400 } -> abort "Bad request",  // Not a failure
-  { code: 500 } -> abort "Server error",  // This counts as failure
-  _ -> continue
-}
-```