reqon-dsl 0.2.0 → 0.3.0

package/docusaurus/docs/http/pagination.md

@@ -1,361 +0,0 @@
----
-sidebar_position: 2
----
-
-# Pagination
-
-Reqon provides built-in support for the three most common pagination strategies: offset-based, page number-based, and cursor-based.
-
-## Pagination Strategies
-
-### Offset-Based Pagination
-
-Uses an offset value that increments by page size:
-
-```vague
-get "/users" {
-  paginate: offset(offset, 100),
-  until: length(response.data) == 0
-}
-```
-
-How it works:
-- First request: `?offset=0`
-- Second request: `?offset=100`
-- Third request: `?offset=200`
-- ...continues until `until` condition is true
-
-Parameters:
-- `offset` - Query parameter name for the offset value
-- `100` - Page size (items per request)
-
-### Page Number-Based Pagination
-
-Uses a page number starting from 1:
-
-```vague
-get "/users" {
-  paginate: page(page, 50),
-  until: response.meta.hasNext == false
-}
-```
-
-How it works:
-- First request: `?page=1`
-- Second request: `?page=2`
-- Third request: `?page=3`
-- ...continues until `until` condition is true
-
-Parameters:
-- `page` - Query parameter name
-- `50` - Page size
-
-### Cursor-Based Pagination
-
-Uses a cursor token from the previous response:
-
-```vague
-get "/users" {
-  paginate: cursor(cursor, 100, "meta.nextCursor"),
-  until: response.meta.nextCursor == null
-}
-```
-
-How it works:
-- First request: no cursor parameter
-- Response: `{ data: [...], meta: { nextCursor: "abc123" } }`
-- Second request: `?cursor=abc123`
-- ...continues until cursor is null
-
-Parameters:
-- `cursor` - Query parameter name
-- `100` - Page size
-- `"meta.nextCursor"` - Path to next cursor in response
-
-## Termination Conditions
-
-The `until` option specifies when to stop paginating:
-
-### Empty Response
-
-```vague
-get "/items" {
-  paginate: offset(skip, 100),
-  until: length(response) == 0
-}
-```
-
-### Empty Data Array
-
-```vague
-get "/items" {
-  paginate: offset(skip, 100),
-  until: length(response.data) == 0
-}
-```
-
-### Boolean Flag
-
-```vague
-get "/items" {
-  paginate: page(p, 50),
-  until: response.hasMore == false
-}
-
-// Or
-get "/items" {
-  paginate: page(p, 50),
-  until: response.pagination.hasNext == false
-}
-```
-
-### Null Cursor
-
-```vague
-get "/items" {
-  paginate: cursor(after, 100, "cursor.next"),
-  until: response.cursor.next == null
-}
-```
-
-### Maximum Pages
-
-```vague
-get "/items" {
-  paginate: page(p, 100),
-  until: length(response.items) == 0 or p > 10
-}
-```
-
-### Item Count Threshold
-
-```vague
-get "/items" {
-  paginate: offset(skip, 100),
-  until: length(response) < 100  // Less than full page
-}
-```
-
-## Combining with Other Options
-
-### With Query Parameters
-
-```vague
-get "/users" {
-  params: {
-    status: "active",
-    sort: "created_at"
-  },
-  paginate: offset(offset, 100),
-  until: length(response.users) == 0
-}
-```
-
-### With Retry
-
-```vague
-get "/users" {
-  paginate: cursor(cursor, 100, "nextCursor"),
-  until: response.nextCursor == null,
-  retry: {
-    maxAttempts: 3,
-    backoff: exponential
-  }
-}
-```
-
-### With Incremental Sync
-
-```vague
-get "/users" {
-  paginate: page(page, 100),
-  until: response.hasMore == false,
-  since: lastSync
-}
-```
-
-## Processing Paginated Results
-
-### Accumulative Processing
-
-All pages are accumulated in `response`:
-
-```vague
-action FetchAll {
-  get "/items" {
-    paginate: offset(offset, 100),
-    until: length(response) == 0
-  }
-
-  // response now contains ALL items from all pages
-  for item in response {
-    store item -> items { key: .id }
-  }
-}
-```
-
-### Per-Page Processing
-
-Process each page as it arrives:
-
-```vague
-action ProcessPages {
-  get "/items" {
-    paginate: offset(offset, 100),
-    until: length(response) == 0
-  }
-
-  // For each page fetched, items are accumulated
-  // After pagination completes, process all
-  for item in response {
-    // Each item is processed
-    store item -> items { key: .id }
-  }
-}
-```
-
-## Common API Patterns
-
-### Standard REST API
-
-```vague
-// API: GET /api/users?limit=100&offset=0
-get "/api/users" {
-  params: { limit: 100 },
-  paginate: offset(offset, 100),
-  until: length(response.data) == 0
-}
-```
-
-### GraphQL-Style Cursor
-
-```vague
-// API uses cursor-based pagination
-get "/api/items" {
-  paginate: cursor(after, 50, "pageInfo.endCursor"),
-  until: response.pageInfo.hasNextPage == false
-}
-```
-
-### Link Header Pagination
-
-For APIs using Link headers, use cursor pagination:
-
-```vague
-get "/api/items" {
-  paginate: cursor(page, 100, "links.next"),
-  until: response.links.next == null
-}
-```
-
-## Best Practices
-
-### Choose the Right Strategy
-
-| API Type | Recommended Strategy |
-|----------|---------------------|
-| Stable datasets | Offset or Page |
-| Real-time data | Cursor |
-| Large datasets | Cursor |
-| Simple APIs | Page |
-
-### Handle Partial Pages
-
-```vague
-get "/items" {
-  paginate: offset(offset, 100),
-  until: length(response.items) < 100  // Partial page = last page
-}
-```
-
-### Set Reasonable Page Sizes
-
-```vague
-// Good: reasonable page size
-get "/items" {
-  paginate: offset(offset, 100),
-  until: length(response) == 0
-}
-
-// Avoid: too large (may timeout or exceed limits)
-get "/items" {
-  paginate: offset(offset, 10000),
-  until: length(response) == 0
-}
-
-// Avoid: too small (too many requests)
-get "/items" {
-  paginate: offset(offset, 10),
-  until: length(response) == 0
-}
-```
-
-### Add Safety Limits
-
-```vague
-get "/items" {
-  paginate: page(page, 100),
-  until: length(response.items) == 0 or page > 100  // Max 100 pages
-}
-```
-
-### Combine with Rate Limiting
-
-```vague
-source API {
-  auth: bearer,
-  base: "https://api.example.com",
-  rateLimit: {
-    requestsPerMinute: 60,
-    strategy: "pause"
-  }
-}
-
-action FetchAll {
-  get "/items" {
-    paginate: offset(offset, 100),
-    until: length(response) == 0
-  }
-}
-```
-
-## Troubleshooting
-
-### Infinite Pagination Loop
-
-If pagination never stops:
-
-```vague
-// Add a safety limit
-get "/items" {
-  paginate: page(page, 100),
-  until: length(response.data) == 0 or page > 50  // Stop after 50 pages
-}
-```
-
-### Duplicate Items
-
-Some APIs return overlapping results. Use upsert:
-
-```vague
-get "/items" {
-  paginate: cursor(cursor, 100, "next"),
-  until: response.next == null
-}
-
-for item in response {
-  store item -> items { key: .id, upsert: true }
-}
-```
-
-### Missing Items
-
-If items are being missed, check your termination condition:
-
-```vague
-// May miss items if last page has exactly 100 items
-until: length(response.items) == 0
-
-// Better: check for less than full page
-until: length(response.items) < 100
-```

package/docusaurus/docs/http/rate-limiting.md

@@ -1,383 +0,0 @@
----
-sidebar_position: 5
----
-
-# Rate Limiting
-
-Reqon provides adaptive rate limiting that learns from API responses and respects rate limit headers.
-
-## Source-Level Configuration
-
-```vague
-source API {
-  auth: bearer,
-  base: "https://api.example.com",
-  rateLimit: {
-    requestsPerMinute: 60,
-    strategy: "pause"
-  }
-}
-```
-
-## Rate Limit Options
-
-| Option | Description | Default |
-|--------|-------------|---------|
-| `requestsPerMinute` | Maximum requests per minute | 60 |
-| `strategy` | How to handle limits | `"pause"` |
-| `maxWait` | Maximum wait time (ms) | 60000 |
-
-## Strategies
-
-### Pause Strategy
-
-Wait when rate limit is reached:
-
-```vague
-source API {
-  auth: bearer,
-  base: "https://api.example.com",
-  rateLimit: {
-    requestsPerMinute: 60,
-    strategy: "pause"
-  }
-}
-```
-
-When limit is reached:
-1. Reqon pauses execution
-2. Waits until rate limit window resets
-3. Continues with next request
-
-### Throttle Strategy
-
-Slow down requests proactively:
-
-```vague
-source API {
-  auth: bearer,
-  base: "https://api.example.com",
-  rateLimit: {
-    requestsPerMinute: 60,
-    strategy: "throttle"
-  }
-}
-```
-
-Automatically spaces requests to stay within limits.
-
-### Fail Strategy
-
-Throw error when limit is reached:
-
-```vague
-source API {
-  auth: bearer,
-  base: "https://api.example.com",
-  rateLimit: {
-    requestsPerMinute: 60,
-    strategy: "fail"
-  }
-}
-```
-
-Use with error handling:
-
-```vague
-action FetchWithRateLimitHandling {
-  get "/data"
-
-  match response {
-    { error: "rate_limit" } -> retry { delay: 60000 },
-    _ -> continue
-  }
-}
-```
-
-## Response Header Support
-
-Reqon automatically reads standard rate limit headers:
-
-| Header | Description |
-|--------|-------------|
-| `X-RateLimit-Limit` | Maximum requests allowed |
-| `X-RateLimit-Remaining` | Requests remaining in window |
-| `X-RateLimit-Reset` | When the window resets |
-| `Retry-After` | Seconds to wait before retrying |
-
-### Header Parsing
-
-```http
-HTTP/1.1 429 Too Many Requests
-X-RateLimit-Limit: 100
-X-RateLimit-Remaining: 0
-X-RateLimit-Reset: 1705752000
-Retry-After: 60
-```
-
-Reqon will automatically:
-1. Pause for 60 seconds (from `Retry-After`)
-2. Update internal limit tracking
-3. Retry the request
-
-## Adaptive Rate Limiting
-
-Reqon learns from API responses:
-
-```vague
-source API {
-  auth: bearer,
-  base: "https://api.example.com",
-  rateLimit: {
-    requestsPerMinute: 100,  // Initial estimate
-    strategy: "pause",
-    adaptive: true  // Learn from responses
-  }
-}
-```
-
-With `adaptive: true`:
-- Reqon monitors response headers
-- Adjusts request pacing dynamically
-- Backs off before hitting limits
-
-## Per-Endpoint Rate Limits
-
-Some APIs have different limits per endpoint:
-
-```vague
-mission APISync {
-  source API {
-    auth: bearer,
-    base: "https://api.example.com",
-    rateLimit: { requestsPerMinute: 100 }
-  }
-
-  action FetchUsers {
-    // Standard endpoint - uses default limit
-    get "/users"
-  }
-
-  action FetchReports {
-    // Heavy endpoint - add delay
-    get "/reports" {
-      rateLimit: { requestsPerMinute: 10 }
-    }
-  }
-}
-```
-
-## Combining with Pagination
-
-```vague
-get "/items" {
-  paginate: offset(offset, 100),
-  until: length(response.items) == 0
-}
-```
-
-Rate limiting applies to each page request, not just the action.
-
-## Combining with Retry
-
-```vague
-source API {
-  auth: bearer,
-  base: "https://api.example.com",
-  rateLimit: {
-    requestsPerMinute: 60,
-    strategy: "pause"
-  }
-}
-
-action Fetch {
-  get "/data" {
-    retry: {
-      maxAttempts: 5,
-      backoff: exponential
-    }
-  }
-}
-```
-
-Order of operations:
-1. Rate limiter checks if request is allowed
-2. If not, pauses (based on strategy)
-3. Request is made
-4. If fails, retry logic kicks in
-
-## Handling 429 Responses
-
-Even with rate limiting, you might hit limits. Handle gracefully:
-
-```vague
-action RobustFetch {
-  get "/data"
-
-  match response {
-    { code: 429 } -> retry {
-      maxAttempts: 5,
-      backoff: exponential,
-      initialDelay: 60000  // Wait 1 minute
-    },
-    _ -> continue
-  }
-}
-```
-
-## Multiple Sources with Different Limits
-
-```vague
-mission MultiSourceSync {
-  source HighVolumeAPI {
-    auth: bearer,
-    base: "https://high-volume.api.com",
-    rateLimit: { requestsPerMinute: 1000 }
-  }
-
-  source LowVolumeAPI {
-    auth: bearer,
-    base: "https://limited.api.com",
-    rateLimit: { requestsPerMinute: 10 }
-  }
-
-  action FetchBoth {
-    // These respect their respective limits
-    get HighVolumeAPI "/items"
-    get LowVolumeAPI "/items"
-  }
-}
-```
-
-## Monitoring Rate Limits
-
-Track rate limit status:
-
-```vague
-action MonitoredFetch {
-  get "/data"
-
-  match response {
-    { code: 429, headers: h } -> {
-      store {
-        endpoint: "/data",
-        hitLimit: true,
-        retryAfter: h["Retry-After"],
-        timestamp: now()
-      } -> rateLimitLogs
-      retry { delay: h["Retry-After"] * 1000 }
-    },
-    _ -> continue
-  }
-}
-```
-
-## Best Practices
-
-### Start Conservative
-
-```vague
-// Good: start below the actual limit
-source API {
-  rateLimit: { requestsPerMinute: 50 }  // API allows 60
-}
-
-// Risky: at or above the limit
-source API {
-  rateLimit: { requestsPerMinute: 60 }  // Exactly at limit
-}
-```
-
-### Use Pause for Critical Syncs
-
-```vague
-source API {
-  rateLimit: {
-    requestsPerMinute: 60,
-    strategy: "pause"  // Ensures completion
-  }
-}
-```
-
-### Use Throttle for Background Jobs
-
-```vague
-source API {
-  rateLimit: {
-    requestsPerMinute: 60,
-    strategy: "throttle"  // Smooth, predictable pacing
-  }
-}
-```
-
-### Set Reasonable maxWait
-
-```vague
-source API {
-  rateLimit: {
-    requestsPerMinute: 60,
-    strategy: "pause",
-    maxWait: 300000  // 5 minutes max wait
-  }
-}
-```
-
-### Combine with Circuit Breaker
-
-```vague
-source API {
-  auth: bearer,
-  base: "https://api.example.com",
-  rateLimit: {
-    requestsPerMinute: 60,
-    strategy: "pause"
-  },
-  circuitBreaker: {
-    failureThreshold: 5,
-    resetTimeout: 30000
-  }
-}
-```
-
-## Troubleshooting
-
-### Still Hitting Rate Limits
-
-Lower your configured limit:
-
-```vague
-source API {
-  rateLimit: {
-    requestsPerMinute: 30,  // Lower than API limit
-    strategy: "pause"
-  }
-}
-```
-
-### Requests Too Slow
-
-Check if throttle strategy is too aggressive:
-
-```vague
-// If using throttle, switch to pause
-source API {
-  rateLimit: {
-    requestsPerMinute: 60,
-    strategy: "pause"  // Only waits when needed
-  }
-}
-```
-
-### Inconsistent API Limits
-
-Use adaptive mode:
-
-```vague
-source API {
-  rateLimit: {
-    requestsPerMinute: 60,
-    strategy: "pause",
-    adaptive: true
-  }
-}
-```