reqon-dsl 0.2.0

package/CLAUDE.md

@@ -0,0 +1,72 @@
+# Reqon
+
+A declarative DSL framework for fetch, map, validate pipelines - built on [Vague](https://github.com/mcclowes/vague).
+
+## Architecture
+
+- **Vague** is the DSL layer (lexer, parser, expression syntax)
+- **Reqon** is the runtime/framework extending Vague with execution semantics
+
+## Project Structure
+
+```
+src/
+├── ast/           # Extended AST nodes (missions, actions, steps)
+├── auth/          # Rate limiting and authentication providers
+├── errors/        # Structured error classes (ParseError, RuntimeError, etc.)
+├── execution/     # Execution state management and persistence
+├── interpreter/   # Runtime execution
+│   ├── context.ts       # Execution context (stores, variables)
+│   ├── evaluator.ts     # Expression evaluation
+│   ├── executor.ts      # Mission/action execution
+│   ├── fetch-handler.ts # HTTP fetch with sync checkpoints
+│   ├── pagination.ts    # Pagination strategies (offset, page, cursor)
+│   └── http.ts          # HTTP client with retry/backoff
+├── lexer/         # Reqon keywords (uses Vague's lexer via plugin)
+├── oas/           # OpenAPI spec integration
+├── parser/        # Parser for mission/action/fetch/store syntax
+├── scheduler/     # Cron scheduling for missions
+├── stores/        # Store adapters (memory, file, extensible to SQL/NoSQL)
+├── sync/          # Incremental sync checkpointing
+├── utils/         # Shared utilities (sleep, path traversal, logger)
+├── index.ts       # Main exports
+└── cli.ts         # CLI entry point
+```
+
+## Commands
+
+```bash
+npm run build      # Compile TypeScript
+npm run test       # Run tests in watch mode
+npm run test:run   # Run tests once
+npm run dev        # Watch mode compilation
+```
+
+## DSL Syntax
+
+Key constructs:
+- `mission` - Pipeline definition
+- `source` - API source with auth (oauth2, bearer, basic, api_key), or from OAS spec
+- `store` - Storage target (memory, file, sql, nosql)
+- `action` - Discrete pipeline step
+- `fetch` - HTTP request with optional pagination/retry, or OAS operationId reference
+- `for...in...where` - Iteration with filtering
+- `map...->` - Schema transformation
+- `validate` - Constraint checking with `assume`
+- `run...then` - Pipeline sequencing (supports `run [A, B] then C` for parallel)
+- `match` - Pattern matching (from Vague)
+- `since: lastSync` - Incremental sync with checkpointing
+
+## Code Conventions
+
+- TypeScript with strict mode
+- Vitest for testing
+- Test files alongside implementation (`*.test.ts`)
+- Vague dependency via local file link (`file:../vague`)
+
+## Key Decisions
+
+1. **Extends Vague**: Reqon uses Vague's lexer (via plugin system) and expression syntax; parser extends Vague's token types
+2. **Keyword conflicts**: Parser explicitly handles Reqon keywords (key, partial, upsert, page, etc.) when they appear in option contexts
+3. **`response` identifier**: Special-cased in evaluator to reference `ctx.response`
+4. **Store adapters**: Interface-based design for pluggable storage backends

package/CONTRIBUTING.md

@@ -0,0 +1,161 @@
+# Contributing to Reqon
+
+Thank you for your interest in contributing to Reqon! This guide will help you get started.
+
+## Getting Started
+
+### Prerequisites
+
+- Node.js 18 or higher
+- npm
+- [Vague](https://github.com/mcclowes/vague) cloned as a sibling directory (required for local development)
+
+### Setup
+
+1. Clone the repository:
+   ```bash
+   git clone https://github.com/mcclowes/reqon.git
+   cd reqon
+   ```
+
+2. Clone Vague as a sibling directory:
+   ```bash
+   cd ..
+   git clone https://github.com/mcclowes/vague.git
+   cd reqon
+   ```
+
+3. Install dependencies:
+   ```bash
+   npm install
+   ```
+
+4. Build the project:
+   ```bash
+   npm run build
+   ```
+
+5. Run tests to verify setup:
+   ```bash
+   npm run test:run
+   ```
+
+## Development Workflow
+
+### Available Scripts
+
+| Command | Description |
+|---------|-------------|
+| `npm run build` | Compile TypeScript to `dist/` |
+| `npm run dev` | Watch mode compilation |
+| `npm run test` | Run tests in watch mode |
+| `npm run test:run` | Run tests once |
+| `npm run test:coverage` | Run tests with coverage report |
+
+### Project Structure
+
+```
+src/
+├── ast/           # Extended AST nodes (missions, actions, steps)
+├── auth/          # Rate limiting and authentication providers
+├── errors/        # Structured error classes
+├── execution/     # Execution state management and persistence
+├── interpreter/   # Runtime execution (context, evaluator, executor)
+├── lexer/         # Reqon keywords (uses Vague's lexer via plugin)
+├── oas/           # OpenAPI spec integration
+├── parser/        # Parser for mission/action/fetch/store syntax
+├── scheduler/     # Cron scheduling for missions
+├── stores/        # Store adapters (memory, file, postgrest)
+├── sync/          # Incremental sync checkpointing
+├── utils/         # Shared utilities
+├── index.ts       # Main exports
+└── cli.ts         # CLI entry point
+```
+
+## Code Conventions
+
+### TypeScript
+
+- Strict mode is enabled
+- Use explicit types for function parameters and return values
+- Prefer `interface` over `type` for object shapes
+
+### Testing
+
+- Tests use [Vitest](https://vitest.dev/)
+- Test files are co-located with implementation: `feature.ts` → `feature.test.ts`
+- Write tests for new functionality
+- Ensure existing tests pass before submitting
+
+Example test structure:
+```typescript
+import { describe, it, expect } from 'vitest';
+
+describe('FeatureName', () => {
+  it('should do something specific', () => {
+    // Arrange
+    // Act
+    // Assert
+    expect(result).toBe(expected);
+  });
+});
+```
+
+### Architecture
+
+Reqon extends [Vague](https://github.com/mcclowes/vague), which provides the core DSL layer (lexer, parser, expression syntax). Reqon adds:
+
+- Mission/action/step AST nodes
+- HTTP fetch with pagination and retry
+- Store adapters for persistence
+- Execution context and runtime
+
+When adding features:
+- Extend Vague's lexer/parser if adding new expression syntax
+- Add new step types in `src/ast/` and handle them in `src/interpreter/executor.ts`
+- New store backends implement the `StoreAdapter` interface in `src/stores/`
+
+## Submitting Changes
+
+### Pull Requests
+
+1. Fork the repository
+2. Create a feature branch from `main`
+3. Make your changes
+4. Ensure tests pass: `npm run test:run`
+5. Ensure the build succeeds: `npm run build`
+6. Submit a pull request
+
+### Commit Messages
+
+Write clear, concise commit messages that describe what changed and why:
+
+```
+Add cursor-based pagination support
+
+- Implement cursor pagination strategy in pagination.ts
+- Add cursor option to fetch step parser
+- Add tests for cursor pagination
+```
+
+### Code Review
+
+All submissions require review. We aim to provide feedback within a few days.
+
+## Reporting Issues
+
+When reporting bugs, please include:
+
+- Reqon version
+- Node.js version
+- Minimal reproduction case (ideally a `.vague` snippet)
+- Expected vs actual behavior
+- Error messages and stack traces
+
+## Questions?
+
+Open an issue for questions about contributing or the codebase architecture.
+
+## License
+
+By contributing, you agree that your contributions will be licensed under the ISC License.

package/README.md

@@ -0,0 +1,235 @@
+# Reqon
+
+A declarative DSL for fetch, map, validate pipelines - built on [Vague](https://github.com/mcclowes/vague).
+
+## What is Reqon?
+
+Reqon lets you define data synchronization pipelines in a readable, declarative language. Think of it like Temporal.io, but with a focus on API data fetching and transformation.
+
+## Example
+
+```vague
+mission SyncXeroInvoices {
+  source Xero {
+    auth: oauth2,
+    base: "https://api.xero.com/api.xro/2.0"
+  }
+
+  store invoices: memory("invoices")
+  store normalized: memory("normalized")
+
+  action FetchInvoices {
+    get "/Invoices" {
+      paginate: offset(page, 100),
+      until: length(response.Invoices) == 0
+    }
+
+    store response.Invoices -> invoices {
+      key: .InvoiceID,
+      partial: true
+    }
+  }
+
+  action NormalizeInvoices {
+    for invoice in invoices {
+      map invoice -> StandardInvoice {
+        id: .InvoiceID,
+        amount: .Total,
+        status: match .Status {
+          "PAID" => "paid",
+          "AUTHORISED" => "approved",
+          _ => "pending"
+        }
+      }
+
+      validate response {
+        assume .amount >= 0
+      }
+
+      store response -> normalized { key: .id }
+    }
+  }
+
+  run FetchInvoices then NormalizeInvoices
+}
+```
+
+## Installation
+
+```bash
+npm install reqon
+```
+
+## Usage
+
+### CLI
+
+```bash
+reqon sync-invoices.vague --verbose
+reqon sync-invoices.vague --auth ./credentials.json
+reqon sync-invoices.vague --dry-run
+```
+
+### Programmatic
+
+```typescript
+import { parse, execute } from 'reqon';
+
+const program = parse(`
+  mission Test {
+    source API { auth: bearer, base: "https://api.example.com" }
+    store items: memory("items")
+    action Fetch {
+      get "/items"
+      store response -> items { key: .id }
+    }
+    run Fetch
+  }
+`);
+
+const result = await execute(source, {
+  auth: { API: { type: 'bearer', token: 'your-token' } }
+});
+
+console.log(result.stores.get('items').list());
+```
+
+## DSL Reference
+
+### Sources
+
+Sources can be defined with explicit base URLs or by referencing an OpenAPI spec:
+
+```vague
+// Traditional: explicit base URL
+source Name {
+  auth: oauth2 | bearer | basic | api_key,
+  base: "https://api.example.com"
+}
+
+// OAS-based: load from OpenAPI spec (base URL derived from spec)
+source Petstore from "./petstore-openapi.yaml" {
+  auth: bearer,
+  validateResponses: true  // Optional: validate responses against OAS schema
+}
+```
+
+### Stores
+
+```vague
+store name: memory("collection")
+store name: sql("table_name")
+store name: nosql("collection")
+```
+
+### Actions
+
+```vague
+action Name {
+  // Steps: get/post/put/patch/delete, call, for, map, validate, store
+}
+```
+
+### HTTP Requests
+
+Two styles are supported:
+
+```vague
+// Traditional: explicit HTTP method and path
+get "/path" {
+  paginate: offset(page, 100),
+  until: response.items.length == 0,
+  retry: { maxAttempts: 3, backoff: exponential }
+}
+
+// OAS-based: reference by Source.operationId
+call Petstore.listPets {
+  paginate: cursor(cursor, 20, "nextCursor"),
+  until: response.pets.length == 0
+}
+```
+
+When using OAS-based `call`, the HTTP method and path are resolved from the OpenAPI spec automatically.
+
+### Iteration
+
+```vague
+for item in collection where .status == "pending" {
+  // nested steps
+}
+```
+
+### Mapping
+
+```vague
+map source -> TargetSchema {
+  field: .sourceField,
+  computed: .price * .quantity,
+  status: match .state {
+    "A" => "active",
+    _ => "unknown"
+  }
+}
+```
+
+### Validation
+
+```vague
+validate target {
+  assume .amount > 0,
+  assume .date >= .createdAt
+}
+```
+
+### Pipeline
+
+```vague
+// Sequential execution
+run Step1 then Step2 then Step3
+
+// Parallel execution with brackets
+run [Step1, Step2] then Step3  // Step1 and Step2 run in parallel, then Step3
+```
+
+## OpenAPI Integration
+
+Reqon can consume OpenAPI specs directly, eliminating the need for handwritten SDK code:
+
+```vague
+mission SyncPets {
+  // Load API definition from OpenAPI spec
+  source Petstore from "./petstore.yaml" {
+    auth: bearer,
+    validateResponses: true
+  }
+
+  store pets: memory("pets")
+
+  action FetchPets {
+    // Use operationId from spec - method and path are resolved automatically
+    call Petstore.listPets
+
+    store response.pets -> pets { key: .id }
+  }
+
+  run FetchPets
+}
+```
+
+Benefits:
+- **No SDK required** - The OpenAPI spec *is* the SDK
+- **Always up-to-date** - Spec changes are picked up automatically
+- **Response validation** - Validate API responses against the spec's schemas
+- **Auto-discovery** - Base URLs, paths, and methods come from the spec
+
+## Development
+
+```bash
+npm run build      # Compile TypeScript
+npm run test:run   # Run tests
+npm run dev        # Watch mode
+```
+
+## License
+
+ISC

package/TODO.md

@@ -0,0 +1,51 @@
+# TODO
+
+## Core Features
+
+- [x] **State persistence** - Durable execution state for resumable missions with checkpointing
+- [x] **Incremental sync** - `since: lastSync` parameter for "only fetch changed since last run"
+- [ ] **Idempotency** - Upsert semantics and conflict resolution strategies
+- [x] **Schema overloading** - `match response { Schema1 -> ..., Schema2 -> ... }` - auto-fork based on response shape matching
+- [x] **Error handling via match** - Flow control directives (continue, skip, abort, retry, queue, jump) in match arms
+- [x] **Multi-file missions** - Split actions into separate files within a folder (mission.vague + action files)
+- [x] **Vague plugin system** - Extended Vague with runtime-extensible keywords and statement parsers; Reqon exports a plugin for Vague integration
+
+## Store Adapters
+
+- [x] **File adapter** - JSON file-based storage in `.reqon-data/` for local development
+- [ ] **SQL adapter** - PostgreSQL/MySQL store implementation
+- [ ] **NoSQL adapter** - MongoDB/DynamoDB store implementation
+
+## API Integration
+
+- [x] **OpenAPI integration** - Load sources from OAS, resolve operationIds, validate responses
+- [x] **Rate limiting** - Adaptive rate limiter, parses X-RateLimit-* headers, respects Retry-After, supports pause/throttle/fail strategies with callbacks
+- [x] **OAuth2 flow** - Token store interface, auto-refresh before expiry, 401 retry with refresh
+- [ ] **Connection registry** - Multi-tenant token management with proactive refresh
+
+## DSL Enhancements
+
+- [x] **`is` type checking** - `assume .items is array` syntax
+- [x] **Parallel execution** - `run [Step1, Step2] then Step3` (bracket syntax for parallel stages)
+- [x] **Conditional actions** - `run Step1 then Step2 if condition` (already implemented in parser)
+- [x] **Variables/let bindings** - Reusable values within missions
+- [x] **Schema definitions** - Full Vague schema support with validation and matching
+
+## Developer Experience
+
+- [x] **Better error messages** - Line numbers, column, source context with pointer in parse/lexer/runtime errors
+- [ ] **VS Code extension** - Syntax highlighting and LSP for `.vague` files
+- [ ] **Debug mode** - Step-through execution with state inspection
+- [ ] **Dry run improvements** - Mock responses based on schema
+
+## Testing
+
+- [ ] **Integration tests with real APIs** - Xero sandbox, etc.
+- [ ] **Property-based testing** - Fuzzing the parser
+- [ ] **Benchmark suite** - Performance testing for large datasets
+
+## Documentation
+
+- [ ] **More examples** - QuickBooks, Stripe, Shopify integrations
+- [ ] **Architecture docs** - How Reqon extends Vague
+- [ ] **Contributing guide**

package/dist/ast/index.d.ts

@@ -0,0 +1 @@
+export * from './nodes.js';

package/dist/ast/index.js

@@ -0,0 +1 @@
+export * from './nodes.js';