reqon-dsl 0.2.0 → 0.3.0

package/CLAUDE.md

@@ -1,72 +0,0 @@
-# Reqon
-
-A declarative DSL framework for fetch, map, validate pipelines - built on [Vague](https://github.com/mcclowes/vague).
-
-## Architecture
-
-- **Vague** is the DSL layer (lexer, parser, expression syntax)
-- **Reqon** is the runtime/framework extending Vague with execution semantics
-
-## Project Structure
-
-```
-src/
-├── ast/           # Extended AST nodes (missions, actions, steps)
-├── auth/          # Rate limiting and authentication providers
-├── errors/        # Structured error classes (ParseError, RuntimeError, etc.)
-├── execution/     # Execution state management and persistence
-├── interpreter/   # Runtime execution
-│   ├── context.ts       # Execution context (stores, variables)
-│   ├── evaluator.ts     # Expression evaluation
-│   ├── executor.ts      # Mission/action execution
-│   ├── fetch-handler.ts # HTTP fetch with sync checkpoints
-│   ├── pagination.ts    # Pagination strategies (offset, page, cursor)
-│   └── http.ts          # HTTP client with retry/backoff
-├── lexer/         # Reqon keywords (uses Vague's lexer via plugin)
-├── oas/           # OpenAPI spec integration
-├── parser/        # Parser for mission/action/fetch/store syntax
-├── scheduler/     # Cron scheduling for missions
-├── stores/        # Store adapters (memory, file, extensible to SQL/NoSQL)
-├── sync/          # Incremental sync checkpointing
-├── utils/         # Shared utilities (sleep, path traversal, logger)
-├── index.ts       # Main exports
-└── cli.ts         # CLI entry point
-```
-
-## Commands
-
-```bash
-npm run build      # Compile TypeScript
-npm run test       # Run tests in watch mode
-npm run test:run   # Run tests once
-npm run dev        # Watch mode compilation
-```
-
-## DSL Syntax
-
-Key constructs:
-- `mission` - Pipeline definition
-- `source` - API source with auth (oauth2, bearer, basic, api_key), or from OAS spec
-- `store` - Storage target (memory, file, sql, nosql)
-- `action` - Discrete pipeline step
-- `fetch` - HTTP request with optional pagination/retry, or OAS operationId reference
-- `for...in...where` - Iteration with filtering
-- `map...->` - Schema transformation
-- `validate` - Constraint checking with `assume`
-- `run...then` - Pipeline sequencing (supports `run [A, B] then C` for parallel)
-- `match` - Pattern matching (from Vague)
-- `since: lastSync` - Incremental sync with checkpointing
-
-## Code Conventions
-
-- TypeScript with strict mode
-- Vitest for testing
-- Test files alongside implementation (`*.test.ts`)
-- Vague dependency via local file link (`file:../vague`)
-
-## Key Decisions
-
-1. **Extends Vague**: Reqon uses Vague's lexer (via plugin system) and expression syntax; parser extends Vague's token types
-2. **Keyword conflicts**: Parser explicitly handles Reqon keywords (key, partial, upsert, page, etc.) when they appear in option contexts
-3. **`response` identifier**: Special-cased in evaluator to reference `ctx.response`
-4. **Store adapters**: Interface-based design for pluggable storage backends

package/CONTRIBUTING.md

@@ -1,161 +0,0 @@
-# Contributing to Reqon
-
-Thank you for your interest in contributing to Reqon! This guide will help you get started.
-
-## Getting Started
-
-### Prerequisites
-
-- Node.js 18 or higher
-- npm
-- [Vague](https://github.com/mcclowes/vague) cloned as a sibling directory (required for local development)
-
-### Setup
-
-1. Clone the repository:
-   ```bash
-   git clone https://github.com/mcclowes/reqon.git
-   cd reqon
-   ```
-
-2. Clone Vague as a sibling directory:
-   ```bash
-   cd ..
-   git clone https://github.com/mcclowes/vague.git
-   cd reqon
-   ```
-
-3. Install dependencies:
-   ```bash
-   npm install
-   ```
-
-4. Build the project:
-   ```bash
-   npm run build
-   ```
-
-5. Run tests to verify setup:
-   ```bash
-   npm run test:run
-   ```
-
-## Development Workflow
-
-### Available Scripts
-
-| Command | Description |
-|---------|-------------|
-| `npm run build` | Compile TypeScript to `dist/` |
-| `npm run dev` | Watch mode compilation |
-| `npm run test` | Run tests in watch mode |
-| `npm run test:run` | Run tests once |
-| `npm run test:coverage` | Run tests with coverage report |
-
-### Project Structure
-
-```
-src/
-├── ast/           # Extended AST nodes (missions, actions, steps)
-├── auth/          # Rate limiting and authentication providers
-├── errors/        # Structured error classes
-├── execution/     # Execution state management and persistence
-├── interpreter/   # Runtime execution (context, evaluator, executor)
-├── lexer/         # Reqon keywords (uses Vague's lexer via plugin)
-├── oas/           # OpenAPI spec integration
-├── parser/        # Parser for mission/action/fetch/store syntax
-├── scheduler/     # Cron scheduling for missions
-├── stores/        # Store adapters (memory, file, postgrest)
-├── sync/          # Incremental sync checkpointing
-├── utils/         # Shared utilities
-├── index.ts       # Main exports
-└── cli.ts         # CLI entry point
-```
-
-## Code Conventions
-
-### TypeScript
-
-- Strict mode is enabled
-- Use explicit types for function parameters and return values
-- Prefer `interface` over `type` for object shapes
-
-### Testing
-
-- Tests use [Vitest](https://vitest.dev/)
-- Test files are co-located with implementation: `feature.ts` → `feature.test.ts`
-- Write tests for new functionality
-- Ensure existing tests pass before submitting
-
-Example test structure:
-```typescript
-import { describe, it, expect } from 'vitest';
-
-describe('FeatureName', () => {
-  it('should do something specific', () => {
-    // Arrange
-    // Act
-    // Assert
-    expect(result).toBe(expected);
-  });
-});
-```
-
-### Architecture
-
-Reqon extends [Vague](https://github.com/mcclowes/vague), which provides the core DSL layer (lexer, parser, expression syntax). Reqon adds:
-
-- Mission/action/step AST nodes
-- HTTP fetch with pagination and retry
-- Store adapters for persistence
-- Execution context and runtime
-
-When adding features:
-- Extend Vague's lexer/parser if adding new expression syntax
-- Add new step types in `src/ast/` and handle them in `src/interpreter/executor.ts`
-- New store backends implement the `StoreAdapter` interface in `src/stores/`
-
-## Submitting Changes
-
-### Pull Requests
-
-1. Fork the repository
-2. Create a feature branch from `main`
-3. Make your changes
-4. Ensure tests pass: `npm run test:run`
-5. Ensure the build succeeds: `npm run build`
-6. Submit a pull request
-
-### Commit Messages
-
-Write clear, concise commit messages that describe what changed and why:
-
-```
-Add cursor-based pagination support
-
-- Implement cursor pagination strategy in pagination.ts
-- Add cursor option to fetch step parser
-- Add tests for cursor pagination
-```
-
-### Code Review
-
-All submissions require review. We aim to provide feedback within a few days.
-
-## Reporting Issues
-
-When reporting bugs, please include:
-
-- Reqon version
-- Node.js version
-- Minimal reproduction case (ideally a `.vague` snippet)
-- Expected vs actual behavior
-- Error messages and stack traces
-
-## Questions?
-
-Open an issue for questions about contributing or the codebase architecture.
-
-## License
-
-By contributing, you agree that your contributions will be licensed under the ISC License.

package/TODO.md

@@ -1,51 +0,0 @@
-# TODO
-
-## Core Features
-
-- [x] **State persistence** - Durable execution state for resumable missions with checkpointing
-- [x] **Incremental sync** - `since: lastSync` parameter for "only fetch changed since last run"
-- [ ] **Idempotency** - Upsert semantics and conflict resolution strategies
-- [x] **Schema overloading** - `match response { Schema1 -> ..., Schema2 -> ... }` - auto-fork based on response shape matching
-- [x] **Error handling via match** - Flow control directives (continue, skip, abort, retry, queue, jump) in match arms
-- [x] **Multi-file missions** - Split actions into separate files within a folder (mission.vague + action files)
-- [x] **Vague plugin system** - Extended Vague with runtime-extensible keywords and statement parsers; Reqon exports a plugin for Vague integration
-
-## Store Adapters
-
-- [x] **File adapter** - JSON file-based storage in `.reqon-data/` for local development
-- [ ] **SQL adapter** - PostgreSQL/MySQL store implementation
-- [ ] **NoSQL adapter** - MongoDB/DynamoDB store implementation
-
-## API Integration
-
-- [x] **OpenAPI integration** - Load sources from OAS, resolve operationIds, validate responses
-- [x] **Rate limiting** - Adaptive rate limiter, parses X-RateLimit-* headers, respects Retry-After, supports pause/throttle/fail strategies with callbacks
-- [x] **OAuth2 flow** - Token store interface, auto-refresh before expiry, 401 retry with refresh
-- [ ] **Connection registry** - Multi-tenant token management with proactive refresh
-
-## DSL Enhancements
-
-- [x] **`is` type checking** - `assume .items is array` syntax
-- [x] **Parallel execution** - `run [Step1, Step2] then Step3` (bracket syntax for parallel stages)
-- [x] **Conditional actions** - `run Step1 then Step2 if condition` (already implemented in parser)
-- [x] **Variables/let bindings** - Reusable values within missions
-- [x] **Schema definitions** - Full Vague schema support with validation and matching
-
-## Developer Experience
-
-- [x] **Better error messages** - Line numbers, column, source context with pointer in parse/lexer/runtime errors
-- [ ] **VS Code extension** - Syntax highlighting and LSP for `.vague` files
-- [ ] **Debug mode** - Step-through execution with state inspection
-- [ ] **Dry run improvements** - Mock responses based on schema
-
-## Testing
-
-- [ ] **Integration tests with real APIs** - Xero sandbox, etc.
-- [ ] **Property-based testing** - Fuzzing the parser
-- [ ] **Benchmark suite** - Performance testing for large datasets
-
-## Documentation
-
-- [ ] **More examples** - QuickBooks, Stripe, Shopify integrations
-- [ ] **Architecture docs** - How Reqon extends Vague
-- [ ] **Contributing guide**

package/dist/auth/auth.test.d.ts

@@ -1 +0,0 @@
-export {};

package/dist/auth/auth.test.js

@@ -1,255 +0,0 @@
-import { describe, it, expect, beforeEach, vi } from 'vitest';
-import { AdaptiveRateLimiter, parseRateLimitHeaders, RateLimitError, RateLimitTimeoutError, } from './rate-limiter.js';
-import { InMemoryTokenStore } from './token-store.js';
-describe('parseRateLimitHeaders', () => {
-    it('parses X-RateLimit headers', () => {
-        const headers = {
-            'X-RateLimit-Limit': '100',
-            'X-RateLimit-Remaining': '42',
-            'X-RateLimit-Reset': '1700000000',
-        };
-        const info = parseRateLimitHeaders(headers);
-        expect(info.limit).toBe(100);
-        expect(info.remaining).toBe(42);
-        expect(info.resetAt).toBeInstanceOf(Date);
-        expect(info.resetAt?.getTime()).toBe(1700000000 * 1000);
-    });
-    it('parses lowercase ratelimit headers', () => {
-        const headers = {
-            'ratelimit-limit': '60',
-            'ratelimit-remaining': '5',
-        };
-        const info = parseRateLimitHeaders(headers);
-        expect(info.limit).toBe(60);
-        expect(info.remaining).toBe(5);
-    });
-    it('parses Retry-After header (seconds)', () => {
-        const headers = {
-            'Retry-After': '30',
-        };
-        const info = parseRateLimitHeaders(headers);
-        expect(info.retryAfter).toBe(30);
-    });
-    it('handles missing headers gracefully', () => {
-        const info = parseRateLimitHeaders({});
-        expect(info.limit).toBeUndefined();
-        expect(info.remaining).toBeUndefined();
-        expect(info.resetAt).toBeUndefined();
-        expect(info.retryAfter).toBeUndefined();
-    });
-});
-describe('AdaptiveRateLimiter', () => {
-    let limiter;
-    beforeEach(() => {
-        limiter = new AdaptiveRateLimiter();
-    });
-    it('allows requests when no limits recorded', async () => {
-        const canProceed = await limiter.canProceed('TestAPI');
-        expect(canProceed).toBe(true);
-    });
-    it('blocks requests when remaining is 0', async () => {
-        const futureReset = new Date(Date.now() + 60000);
-        limiter.recordResponse('TestAPI', {
-            remaining: 0,
-            limit: 100,
-            resetAt: futureReset,
-        });
-        const canProceed = await limiter.canProceed('TestAPI');
-        expect(canProceed).toBe(false);
-    });
-    it('allows requests after reset time passes', async () => {
-        const pastReset = new Date(Date.now() - 1000);
-        limiter.recordResponse('TestAPI', {
-            remaining: 0,
-            limit: 100,
-            resetAt: pastReset,
-        });
-        const canProceed = await limiter.canProceed('TestAPI');
-        expect(canProceed).toBe(true);
-    });
-    it('blocks during retry-after period', async () => {
-        limiter.recordResponse('TestAPI', {
-            retryAfter: 60, // 60 seconds
-        });
-        const canProceed = await limiter.canProceed('TestAPI');
-        expect(canProceed).toBe(false);
-    });
-    it('tracks limits per source', async () => {
-        limiter.recordResponse('API1', { remaining: 0, resetAt: new Date(Date.now() + 60000) });
-        limiter.recordResponse('API2', { remaining: 50, limit: 100 });
-        expect(await limiter.canProceed('API1')).toBe(false);
-        expect(await limiter.canProceed('API2')).toBe(true);
-    });
-    it('tracks limits per endpoint within source', async () => {
-        limiter.recordResponse('API', { remaining: 0, resetAt: new Date(Date.now() + 60000) }, '/invoices');
-        limiter.recordResponse('API', { remaining: 50, limit: 100 }, '/contacts');
-        expect(await limiter.canProceed('API', '/invoices')).toBe(false);
-        expect(await limiter.canProceed('API', '/contacts')).toBe(true);
-    });
-    it('provides accurate status', () => {
-        limiter.recordResponse('TestAPI', {
-            remaining: 42,
-            limit: 100,
-            resetAt: new Date(Date.now() + 60000),
-        });
-        const status = limiter.getStatus('TestAPI');
-        expect(status.remaining).toBe(42);
-        expect(status.limit).toBe(100);
-        expect(status.isLimited).toBe(false);
-    });
-    describe('fail strategy', () => {
-        it('throws RateLimitError immediately when rate limited', async () => {
-            limiter.configure('FailAPI', { strategy: 'fail' });
-            limiter.recordResponse('FailAPI', {
-                remaining: 0,
-                resetAt: new Date(Date.now() + 60000),
-            });
-            await expect(limiter.waitForCapacity('FailAPI')).rejects.toThrow(RateLimitError);
-        });
-        it('includes reset time in error', async () => {
-            limiter.configure('FailAPI', { strategy: 'fail' });
-            const resetAt = new Date(Date.now() + 60000);
-            limiter.recordResponse('FailAPI', { remaining: 0, resetAt });
-            try {
-                await limiter.waitForCapacity('FailAPI');
-                expect.fail('Should have thrown');
-            }
-            catch (error) {
-                expect(error).toBeInstanceOf(RateLimitError);
-                expect(error.resetAt).toEqual(resetAt);
-            }
-        });
-    });
-    describe('pause strategy', () => {
-        it('throws RateLimitTimeoutError when maxWait exceeded', async () => {
-            limiter.configure('PauseAPI', { strategy: 'pause', maxWait: 1 });
-            limiter.recordResponse('PauseAPI', {
-                remaining: 0,
-                resetAt: new Date(Date.now() + 60000), // 60s in future
-            });
-            await expect(limiter.waitForCapacity('PauseAPI')).rejects.toThrow(RateLimitTimeoutError);
-        });
-        it('proceeds after reset time passes', async () => {
-            limiter.configure('PauseAPI', { strategy: 'pause', maxWait: 5 });
-            // Reset in 50ms
-            limiter.recordResponse('PauseAPI', {
-                remaining: 0,
-                resetAt: new Date(Date.now() + 50),
-            });
-            // Should complete without throwing
-            await limiter.waitForCapacity('PauseAPI');
-        });
-    });
-    describe('throttle strategy', () => {
-        it('calculates delay based on remaining requests and reset time', () => {
-            limiter.configure('ThrottleAPI', { strategy: 'throttle' });
-            limiter.recordResponse('ThrottleAPI', {
-                remaining: 10,
-                limit: 100,
-                resetAt: new Date(Date.now() + 10000), // 10s left
-            });
-            const delay = limiter.getThrottleDelay('ThrottleAPI');
-            // 10s / 10 remaining = 1s per request = 1000ms
-            expect(delay).toBeGreaterThanOrEqual(900);
-            expect(delay).toBeLessThanOrEqual(1100);
-        });
-        it('returns 0 delay when not in throttle mode', () => {
-            limiter.configure('PauseAPI', { strategy: 'pause' });
-            limiter.recordResponse('PauseAPI', { remaining: 10, limit: 100 });
-            const delay = limiter.getThrottleDelay('PauseAPI');
-            expect(delay).toBe(0);
-        });
-        it('uses fallback RPM when no rate limit headers', () => {
-            limiter.configure('FallbackAPI', { strategy: 'throttle', fallbackRpm: 60 });
-            limiter.recordResponse('FallbackAPI', {}); // No rate limit info
-            const delay = limiter.getThrottleDelay('FallbackAPI');
-            // 60 RPM = 1 per second = 1000ms intervals
-            expect(delay).toBeLessThanOrEqual(1000);
-        });
-    });
-    describe('callbacks', () => {
-        it('calls onRateLimited when rate limited', async () => {
-            const onRateLimited = vi.fn();
-            limiter.setCallbacks({ onRateLimited });
-            // Use maxWait of 10s so it doesn't timeout immediately (wait is only 50ms)
-            limiter.configure('CallbackAPI', { strategy: 'pause', maxWait: 10 });
-            limiter.recordResponse('CallbackAPI', {
-                remaining: 0,
-                resetAt: new Date(Date.now() + 50), // Resets in 50ms
-            });
-            await limiter.waitForCapacity('CallbackAPI');
-            expect(onRateLimited).toHaveBeenCalledTimes(1);
-            expect(onRateLimited).toHaveBeenCalledWith(expect.objectContaining({
-                source: 'CallbackAPI',
-                strategy: 'pause',
-            }));
-        });
-        it('calls onResumed after waiting completes', async () => {
-            const onResumed = vi.fn();
-            limiter.setCallbacks({ onResumed });
-            limiter.configure('ResumeAPI', { strategy: 'pause', maxWait: 5 });
-            // Reset in 50ms
-            limiter.recordResponse('ResumeAPI', {
-                remaining: 0,
-                resetAt: new Date(Date.now() + 50),
-            });
-            await limiter.waitForCapacity('ResumeAPI');
-            expect(onResumed).toHaveBeenCalledTimes(1);
-            expect(onResumed).toHaveBeenCalledWith(expect.objectContaining({
-                source: 'ResumeAPI',
-            }));
-        });
-    });
-});
-describe('InMemoryTokenStore', () => {
-    let store;
-    beforeEach(() => {
-        store = new InMemoryTokenStore();
-    });
-    it('stores and retrieves tokens', async () => {
-        const tokens = {
-            accessToken: 'access123',
-            refreshToken: 'refresh456',
-            expiresAt: new Date(Date.now() + 3600000),
-        };
-        await store.set('connection-1', tokens);
-        const retrieved = await store.get('connection-1');
-        expect(retrieved?.accessToken).toBe('access123');
-        expect(retrieved?.refreshToken).toBe('refresh456');
-    });
-    it('returns null for unknown connections', async () => {
-        const result = await store.get('unknown');
-        expect(result).toBeNull();
-    });
-    it('deletes tokens', async () => {
-        await store.set('connection-1', { accessToken: 'test' });
-        await store.delete('connection-1');
-        const result = await store.get('connection-1');
-        expect(result).toBeNull();
-    });
-    it('lists all connections', async () => {
-        await store.set('conn-1', { accessToken: 'a' });
-        await store.set('conn-2', { accessToken: 'b' });
-        await store.set('conn-3', { accessToken: 'c' });
-        const connections = await store.list();
-        expect(connections).toHaveLength(3);
-        expect(connections).toContain('conn-1');
-        expect(connections).toContain('conn-2');
-        expect(connections).toContain('conn-3');
-    });
-    it('identifies tokens needing refresh', async () => {
-        // Token expiring in 10 seconds (within 5 min buffer)
-        await store.set('expiring-soon', {
-            accessToken: 'test',
-            expiresAt: new Date(Date.now() + 10000),
-        });
-        // Token not expiring soon
-        await store.set('valid', {
-            accessToken: 'test',
-            expiresAt: new Date(Date.now() + 3600000),
-        });
-        const needsRefresh = await store.getTokensNeedingRefresh(300);
-        expect(needsRefresh).toContain('expiring-soon');
-        expect(needsRefresh).not.toContain('valid');
-    });
-});

package/dist/errors/errors.test.d.ts

@@ -1 +0,0 @@
-export {};