reqon-dsl 0.2.0

package/docusaurus/docs/http/pagination.md

@@ -0,0 +1,361 @@
+---
+sidebar_position: 2
+---
+
+# Pagination
+
+Reqon provides built-in support for the three most common pagination strategies: offset-based, page number-based, and cursor-based.
+
+## Pagination Strategies
+
+### Offset-Based Pagination
+
+Uses an offset value that increments by page size:
+
+```vague
+get "/users" {
+  paginate: offset(offset, 100),
+  until: length(response.data) == 0
+}
+```
+
+How it works:
+- First request: `?offset=0`
+- Second request: `?offset=100`
+- Third request: `?offset=200`
+- ...continues until `until` condition is true
+
+Parameters:
+- `offset` - Query parameter name for the offset value
+- `100` - Page size (items per request)
+
+### Page Number-Based Pagination
+
+Uses a page number starting from 1:
+
+```vague
+get "/users" {
+  paginate: page(page, 50),
+  until: response.meta.hasNext == false
+}
+```
+
+How it works:
+- First request: `?page=1`
+- Second request: `?page=2`
+- Third request: `?page=3`
+- ...continues until `until` condition is true
+
+Parameters:
+- `page` - Query parameter name
+- `50` - Page size
+
+### Cursor-Based Pagination
+
+Uses a cursor token from the previous response:
+
+```vague
+get "/users" {
+  paginate: cursor(cursor, 100, "meta.nextCursor"),
+  until: response.meta.nextCursor == null
+}
+```
+
+How it works:
+- First request: no cursor parameter
+- Response: `{ data: [...], meta: { nextCursor: "abc123" } }`
+- Second request: `?cursor=abc123`
+- ...continues until cursor is null
+
+Parameters:
+- `cursor` - Query parameter name
+- `100` - Page size
+- `"meta.nextCursor"` - Path to next cursor in response
+
+## Termination Conditions
+
+The `until` option specifies when to stop paginating:
+
+### Empty Response
+
+```vague
+get "/items" {
+  paginate: offset(skip, 100),
+  until: length(response) == 0
+}
+```
+
+### Empty Data Array
+
+```vague
+get "/items" {
+  paginate: offset(skip, 100),
+  until: length(response.data) == 0
+}
+```
+
+### Boolean Flag
+
+```vague
+get "/items" {
+  paginate: page(p, 50),
+  until: response.hasMore == false
+}
+
+// Or
+get "/items" {
+  paginate: page(p, 50),
+  until: response.pagination.hasNext == false
+}
+```
+
+### Null Cursor
+
+```vague
+get "/items" {
+  paginate: cursor(after, 100, "cursor.next"),
+  until: response.cursor.next == null
+}
+```
+
+### Maximum Pages
+
+```vague
+get "/items" {
+  paginate: page(p, 100),
+  until: length(response.items) == 0 or p > 10
+}
+```
+
+### Item Count Threshold
+
+```vague
+get "/items" {
+  paginate: offset(skip, 100),
+  until: length(response) < 100  // Less than full page
+}
+```
+
+## Combining with Other Options
+
+### With Query Parameters
+
+```vague
+get "/users" {
+  params: {
+    status: "active",
+    sort: "created_at"
+  },
+  paginate: offset(offset, 100),
+  until: length(response.users) == 0
+}
+```
+
+### With Retry
+
+```vague
+get "/users" {
+  paginate: cursor(cursor, 100, "nextCursor"),
+  until: response.nextCursor == null,
+  retry: {
+    maxAttempts: 3,
+    backoff: exponential
+  }
+}
+```
+
+### With Incremental Sync
+
+```vague
+get "/users" {
+  paginate: page(page, 100),
+  until: response.hasMore == false,
+  since: lastSync
+}
+```
+
+## Processing Paginated Results
+
+### Accumulative Processing
+
+All pages are accumulated in `response`:
+
+```vague
+action FetchAll {
+  get "/items" {
+    paginate: offset(offset, 100),
+    until: length(response) == 0
+  }
+
+  // response now contains ALL items from all pages
+  for item in response {
+    store item -> items { key: .id }
+  }
+}
+```
+
+### Per-Page Processing
+
+Process each page as it arrives:
+
+```vague
+action ProcessPages {
+  get "/items" {
+    paginate: offset(offset, 100),
+    until: length(response) == 0
+  }
+
+  // For each page fetched, items are accumulated
+  // After pagination completes, process all
+  for item in response {
+    // Each item is processed
+    store item -> items { key: .id }
+  }
+}
+```
+
+## Common API Patterns
+
+### Standard REST API
+
+```vague
+// API: GET /api/users?limit=100&offset=0
+get "/api/users" {
+  params: { limit: 100 },
+  paginate: offset(offset, 100),
+  until: length(response.data) == 0
+}
+```
+
+### GraphQL-Style Cursor
+
+```vague
+// API uses cursor-based pagination
+get "/api/items" {
+  paginate: cursor(after, 50, "pageInfo.endCursor"),
+  until: response.pageInfo.hasNextPage == false
+}
+```
+
+### Link Header Pagination
+
+For APIs using Link headers, use cursor pagination:
+
+```vague
+get "/api/items" {
+  paginate: cursor(page, 100, "links.next"),
+  until: response.links.next == null
+}
+```
+
+## Best Practices
+
+### Choose the Right Strategy
+
+| API Type | Recommended Strategy |
+|----------|---------------------|
+| Stable datasets | Offset or Page |
+| Real-time data | Cursor |
+| Large datasets | Cursor |
+| Simple APIs | Page |
+
+### Handle Partial Pages
+
+```vague
+get "/items" {
+  paginate: offset(offset, 100),
+  until: length(response.items) < 100  // Partial page = last page
+}
+```
+
+### Set Reasonable Page Sizes
+
+```vague
+// Good: reasonable page size
+get "/items" {
+  paginate: offset(offset, 100),
+  until: length(response) == 0
+}
+
+// Avoid: too large (may timeout or exceed limits)
+get "/items" {
+  paginate: offset(offset, 10000),
+  until: length(response) == 0
+}
+
+// Avoid: too small (too many requests)
+get "/items" {
+  paginate: offset(offset, 10),
+  until: length(response) == 0
+}
+```
+
+### Add Safety Limits
+
+```vague
+get "/items" {
+  paginate: page(page, 100),
+  until: length(response.items) == 0 or page > 100  // Max 100 pages
+}
+```
+
+### Combine with Rate Limiting
+
+```vague
+source API {
+  auth: bearer,
+  base: "https://api.example.com",
+  rateLimit: {
+    requestsPerMinute: 60,
+    strategy: "pause"
+  }
+}
+
+action FetchAll {
+  get "/items" {
+    paginate: offset(offset, 100),
+    until: length(response) == 0
+  }
+}
+```
+
+## Troubleshooting
+
+### Infinite Pagination Loop
+
+If pagination never stops:
+
+```vague
+// Add a safety limit
+get "/items" {
+  paginate: page(page, 100),
+  until: length(response.data) == 0 or page > 50  // Stop after 50 pages
+}
+```
+
+### Duplicate Items
+
+Some APIs return overlapping results. Use upsert:
+
+```vague
+get "/items" {
+  paginate: cursor(cursor, 100, "next"),
+  until: response.next == null
+}
+
+for item in response {
+  store item -> items { key: .id, upsert: true }
+}
+```
+
+### Missing Items
+
+If items are being missed, check your termination condition:
+
+```vague
+// May miss items if last page has exactly 100 items
+until: length(response.items) == 0
+
+// Better: check for less than full page
+until: length(response.items) < 100
+```

package/docusaurus/docs/http/rate-limiting.md

@@ -0,0 +1,383 @@
+---
+sidebar_position: 5
+---
+
+# Rate Limiting
+
+Reqon provides adaptive rate limiting that learns from API responses and respects rate limit headers.
+
+## Source-Level Configuration
+
+```vague
+source API {
+  auth: bearer,
+  base: "https://api.example.com",
+  rateLimit: {
+    requestsPerMinute: 60,
+    strategy: "pause"
+  }
+}
+```
+
+## Rate Limit Options
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| `requestsPerMinute` | Maximum requests per minute | 60 |
+| `strategy` | How to handle limits | `"pause"` |
+| `maxWait` | Maximum wait time (ms) | 60000 |
+
+## Strategies
+
+### Pause Strategy
+
+Wait when rate limit is reached:
+
+```vague
+source API {
+  auth: bearer,
+  base: "https://api.example.com",
+  rateLimit: {
+    requestsPerMinute: 60,
+    strategy: "pause"
+  }
+}
+```
+
+When limit is reached:
+1. Reqon pauses execution
+2. Waits until rate limit window resets
+3. Continues with next request
+
+### Throttle Strategy
+
+Slow down requests proactively:
+
+```vague
+source API {
+  auth: bearer,
+  base: "https://api.example.com",
+  rateLimit: {
+    requestsPerMinute: 60,
+    strategy: "throttle"
+  }
+}
+```
+
+Automatically spaces requests to stay within limits.
+
+### Fail Strategy
+
+Throw error when limit is reached:
+
+```vague
+source API {
+  auth: bearer,
+  base: "https://api.example.com",
+  rateLimit: {
+    requestsPerMinute: 60,
+    strategy: "fail"
+  }
+}
+```
+
+Use with error handling:
+
+```vague
+action FetchWithRateLimitHandling {
+  get "/data"
+
+  match response {
+    { error: "rate_limit" } -> retry { delay: 60000 },
+    _ -> continue
+  }
+}
+```
+
+## Response Header Support
+
+Reqon automatically reads standard rate limit headers:
+
+| Header | Description |
+|--------|-------------|
+| `X-RateLimit-Limit` | Maximum requests allowed |
+| `X-RateLimit-Remaining` | Requests remaining in window |
+| `X-RateLimit-Reset` | When the window resets |
+| `Retry-After` | Seconds to wait before retrying |
+
+### Header Parsing
+
+```http
+HTTP/1.1 429 Too Many Requests
+X-RateLimit-Limit: 100
+X-RateLimit-Remaining: 0
+X-RateLimit-Reset: 1705752000
+Retry-After: 60
+```
+
+Reqon will automatically:
+1. Pause for 60 seconds (from `Retry-After`)
+2. Update internal limit tracking
+3. Retry the request
+
+## Adaptive Rate Limiting
+
+Reqon learns from API responses:
+
+```vague
+source API {
+  auth: bearer,
+  base: "https://api.example.com",
+  rateLimit: {
+    requestsPerMinute: 100,  // Initial estimate
+    strategy: "pause",
+    adaptive: true  // Learn from responses
+  }
+}
+```
+
+With `adaptive: true`:
+- Reqon monitors response headers
+- Adjusts request pacing dynamically
+- Backs off before hitting limits
+
+## Per-Endpoint Rate Limits
+
+Some APIs have different limits per endpoint:
+
+```vague
+mission APISync {
+  source API {
+    auth: bearer,
+    base: "https://api.example.com",
+    rateLimit: { requestsPerMinute: 100 }
+  }
+
+  action FetchUsers {
+    // Standard endpoint - uses default limit
+    get "/users"
+  }
+
+  action FetchReports {
+    // Heavy endpoint - add delay
+    get "/reports" {
+      rateLimit: { requestsPerMinute: 10 }
+    }
+  }
+}
+```
+
+## Combining with Pagination
+
+```vague
+get "/items" {
+  paginate: offset(offset, 100),
+  until: length(response.items) == 0
+}
+```
+
+Rate limiting applies to each page request, not just the action.
+
+## Combining with Retry
+
+```vague
+source API {
+  auth: bearer,
+  base: "https://api.example.com",
+  rateLimit: {
+    requestsPerMinute: 60,
+    strategy: "pause"
+  }
+}
+
+action Fetch {
+  get "/data" {
+    retry: {
+      maxAttempts: 5,
+      backoff: exponential
+    }
+  }
+}
+```
+
+Order of operations:
+1. Rate limiter checks if request is allowed
+2. If not, pauses (based on strategy)
+3. Request is made
+4. If fails, retry logic kicks in
+
+## Handling 429 Responses
+
+Even with rate limiting, you might hit limits. Handle gracefully:
+
+```vague
+action RobustFetch {
+  get "/data"
+
+  match response {
+    { code: 429 } -> retry {
+      maxAttempts: 5,
+      backoff: exponential,
+      initialDelay: 60000  // Wait 1 minute
+    },
+    _ -> continue
+  }
+}
+```
+
+## Multiple Sources with Different Limits
+
+```vague
+mission MultiSourceSync {
+  source HighVolumeAPI {
+    auth: bearer,
+    base: "https://high-volume.api.com",
+    rateLimit: { requestsPerMinute: 1000 }
+  }
+
+  source LowVolumeAPI {
+    auth: bearer,
+    base: "https://limited.api.com",
+    rateLimit: { requestsPerMinute: 10 }
+  }
+
+  action FetchBoth {
+    // These respect their respective limits
+    get HighVolumeAPI "/items"
+    get LowVolumeAPI "/items"
+  }
+}
+```
+
+## Monitoring Rate Limits
+
+Track rate limit status:
+
+```vague
+action MonitoredFetch {
+  get "/data"
+
+  match response {
+    { code: 429, headers: h } -> {
+      store {
+        endpoint: "/data",
+        hitLimit: true,
+        retryAfter: h["Retry-After"],
+        timestamp: now()
+      } -> rateLimitLogs
+      retry { delay: h["Retry-After"] * 1000 }
+    },
+    _ -> continue
+  }
+}
+```
+
+## Best Practices
+
+### Start Conservative
+
+```vague
+// Good: start below the actual limit
+source API {
+  rateLimit: { requestsPerMinute: 50 }  // API allows 60
+}
+
+// Risky: at or above the limit
+source API {
+  rateLimit: { requestsPerMinute: 60 }  // Exactly at limit
+}
+```
+
+### Use Pause for Critical Syncs
+
+```vague
+source API {
+  rateLimit: {
+    requestsPerMinute: 60,
+    strategy: "pause"  // Ensures completion
+  }
+}
+```
+
+### Use Throttle for Background Jobs
+
+```vague
+source API {
+  rateLimit: {
+    requestsPerMinute: 60,
+    strategy: "throttle"  // Smooth, predictable pacing
+  }
+}
+```
+
+### Set Reasonable maxWait
+
+```vague
+source API {
+  rateLimit: {
+    requestsPerMinute: 60,
+    strategy: "pause",
+    maxWait: 300000  // 5 minutes max wait
+  }
+}
+```
+
+### Combine with Circuit Breaker
+
+```vague
+source API {
+  auth: bearer,
+  base: "https://api.example.com",
+  rateLimit: {
+    requestsPerMinute: 60,
+    strategy: "pause"
+  },
+  circuitBreaker: {
+    failureThreshold: 5,
+    resetTimeout: 30000
+  }
+}
+```
+
+## Troubleshooting
+
+### Still Hitting Rate Limits
+
+Lower your configured limit:
+
+```vague
+source API {
+  rateLimit: {
+    requestsPerMinute: 30,  // Lower than API limit
+    strategy: "pause"
+  }
+}
+```
+
+### Requests Too Slow
+
+Check if throttle strategy is too aggressive:
+
+```vague
+// If using throttle, switch to pause
+source API {
+  rateLimit: {
+    requestsPerMinute: 60,
+    strategy: "pause"  // Only waits when needed
+  }
+}
+```
+
+### Inconsistent API Limits
+
+Use adaptive mode:
+
+```vague
+source API {
+  rateLimit: {
+    requestsPerMinute: 60,
+    strategy: "pause",
+    adaptive: true
+  }
+}
+```