ravenflow 0.3.0 → 0.4.0

package/README.md

@@ -1,334 +1,192 @@
 # ravenflow
 
-Type-safe job queue framework for TypeScript, built on [BullMQ](https://docs.bullmq.io/) and [Redis](https://redis.io/).
+Typed background jobs for Node.js, on [BullMQ](https://docs.bullmq.io). Redis holds the work in
+flight; MongoDB holds the backlogs and the history.
 
-Define jobs as plain functions, ravenflow handles scheduling, retries, rate limiting, distributed processing, and crash recovery.
+- **Types end to end.** A send is checked against the job's declaration.
+- **A client and a worker.** Your API sends work, a worker process runs it, and both share one config.
+- **Two triggers.** A clock, and a spool: a backlog in MongoDB drained at the job's pace.
+- **What production asks for.** Chains, deduplication, retries, a rate held across the fleet, levers
+  to pause a job, and a record of every execution.
 
 ## Install
 
 ```bash
 npm install ravenflow
-# or
-pnpm add ravenflow
 ```
 
-**Requirements:** Redis 6+, Node.js 18+. MongoDB required for stateful-etl and token-bucket jobs.
+Requires Node.js 20.19+, Redis and MongoDB. For development:
 
-## Quick Start
-
-```ts
-import {
-  createRavenflowApp,
-  defineConfig,
-  defineSimpleJob,
-} from "ravenflow";
-
-// 1. Define your jobs
-const config = defineConfig({
-  defaultQueue: "default",
-  jobs: {
-    sendEmail: defineSimpleJob({
-      handler: async (job, params: { to: string; subject: string }) => {
-        await sendEmail(params.to, params.subject);
-        return { sent: true };
-      },
-    }),
-  },
-});
-
-// 2. Create the app
-const app = createRavenflowApp(config, {
-  redis: { host: "localhost", port: 6379 },
-});
-
-// 3. Start the worker
-await app.startWorker("default", { concurrency: 10 });
-
-// 4. Send jobs
-await app.jobs.sendEmail.send({ to: "user@example.com", subject: "Hello" });
-
-// Or send and wait for the result
-const result = await app.jobs.sendEmail.sendAndWait({ to: "user@example.com", subject: "Hello" });
-console.log(result); // { sent: true }
+```bash
+docker run -d -p 6379:6379 redis:7
+docker run -d -p 27017:27017 mongo:7
 ```
 
-## Job Types
-
-### Simple Job
+## Create and run a job
 
-One-shot function execution. Ideal for notifications, cleanup tasks, webhooks.
+**1. Declare it.** A job is a handler from a typed payload to a typed result. The config registers
+it under a name.
 
 ```ts
-import { defineSimpleJob } from "ravenflow";
-
-const sendEmail = defineSimpleJob({
-  cron: "*/5 * * * *", // optional: run every 5 minutes
-  handler: async (job, params: { to: string; text: string }) => {
-    const result = await emailService.send(params.to, params.text);
-    return { messageId: result.id };
+// config.ts
+import { defineConfig, defineJob } from 'ravenflow';
+
+export const sendWelcomeEmail = defineJob<{ to: string; name: string }, { messageId: string }>({
+  handler: async ({ params, logger }) => {
+    logger.info({ to: params.to }, 'sending the welcome email');
+    const { id } = await emailTransport.send({ to: params.to, subject: `Welcome, ${params.name}` });
+    return { messageId: id };
   },
 });
-```
-
-### Memory ETL
-
-Extract-Transform-Load pipeline that runs entirely in-memory. Cursor-based pagination, parallel transforms with rate limiting.
-
-```ts
-import { defineMemoryEtlJob } from "ravenflow";
-
-const syncUsers = defineMemoryEtlJob({
-  cron: "0 2 * * *", // daily at 2 AM
-
-  extract: async (cursor: number | null) => {
-    const offset = cursor ?? 0;
-    const users = await api.getUsers({ offset, limit: 100 });
-    return {
-      data: users,
-      cursor: offset + 100,
-      hasMore: users.length === 100,
-    };
-  },
-
-  transform: async (user) => ({
-    id: user.id,
-    email: user.email,
-    normalizedName: user.name.trim().toLowerCase(),
-  }),
 
-  load: async (users) => {
-    await db.users.insertMany(users);
-  },
-
-  steps: {
-    transform: {
-      rateLimit: { perSecond: 50 }, // even spacing: 1 every 20ms
-    },
-  },
+export const config = defineConfig({
+  defaultQueue: 'default',
+  jobs: { sendWelcomeEmail },
 });
 ```
 
-### Distributed ETL
-
-Same extract/transform/load pattern, but transforms are dispatched as independent BullMQ sub-jobs for parallel processing across workers.
+**2. Send it** from any process: your API, a CLI, another job.
 
 ```ts
-import { defineDistributedEtlJob } from "ravenflow";
-
-const processOrders = defineDistributedEtlJob({
-  cron: "*/10 * * * *",
-
-  extract: async (cursor: number | null) => {
-    const orders = await db.orders
-      .find({ status: "pending" })
-      .skip(cursor ?? 0)
-      .limit(50)
-      .toArray();
-    return {
-      data: orders,
-      cursor: (cursor ?? 0) + 50,
-      hasMore: orders.length === 50,
-    };
-  },
-
-  transform: async (order) => {
-    const enriched = await paymentGateway.verify(order.paymentId);
-    return { ...order, verified: enriched.status === "ok" };
-  },
+// client.ts
+import { createRavenflowClient } from 'ravenflow';
+import { config } from './config';
 
-  load: async (orders) => {
-    await db.processedOrders.insertMany(orders);
-  },
-
-  steps: {
-    transform: {
-      rateLimit: { perSecond: 10 },
-      attempts: 3,
-      backoff: { type: "exponential", delay: 1000 },
-    },
-  },
+const client = createRavenflowClient(config, {
+  redis: { host: 'localhost', port: 6379 },
+  mongo: 'mongodb://localhost:27017/myapp',
 });
-```
 
-### Stateful ETL
+await client.jobs.sendWelcomeEmail.send({ to: 'ada@example.com', name: 'Ada' });
+```
 
-Distributed ETL with MongoDB-backed checkpoints and automatic crash recovery. If a worker dies mid-processing, the Stall Watcher detects it and resumes from the last checkpoint.
+**3. Run it** in a worker. A worker serves one queue.
 
 ```ts
-import { defineStatefulEtlJob } from "ravenflow";
-
-const migrateLegacyData = defineStatefulEtlJob({
-  cron: "0 3 * * *",
-  singletonRun: true, // only one run at a time
-  stallTimeoutMs: 60_000, // detect stall after 60s without heartbeat
-  maxStallRetries: 3,
-
-  extract: async (cursor: string | null) => {
-    const page = await legacyApi.fetch({ after: cursor, limit: 100 });
-    return {
-      data: page.items,
-      cursor: page.nextCursor,
-      hasMore: page.hasMore,
-    };
-  },
+// worker.ts
+import { createRavenflowWorker } from 'ravenflow';
+import { config } from './config';
+
+const worker = createRavenflowWorker(config, {
+  redis: { host: 'localhost', port: 6379 },
+  mongo: 'mongodb://localhost:27017/myapp',
+  queue: 'default',
+  concurrency: 10,
+});
 
-  transform: async (item) => {
-    return mapToNewSchema(item);
-  },
+await worker.start();
+process.once('SIGTERM', () => void worker.close());
+```
 
-  load: async (items) => {
-    await newDb.collection("records").insertMany(items);
-  },
+The handler receives `{ params, logger, signal, job, continueWith, deduplicationId }`. What it
+returns is the job's result; what it throws fails the job.
 
-  steps: {
-    transform: {
-      rateLimit: { perSecond: 5 },
-      attempts: 3,
-      backoff: { type: "exponential", delay: 2000 },
-    },
-  },
-});
-```
+A send takes options: `{ delayMs, priority, deduplicationId }`.
 
-**How crash recovery works:**
-1. During processing, the worker sends heartbeats to MongoDB
-2. A `__stall_check` cron job monitors all active runs
-3. If a heartbeat goes stale (exceeds `stallTimeoutMs`), the run is claimed as "stalled"
-4. A `__stall_resume` job re-enqueues the original job with the last checkpoint
-5. Processing resumes from where it left off (no duplicate work)
+## Triggers
 
-## Configuration
+Without a trigger, a client sends the job. A trigger feeds it for you.
 
-### App Options
+### A clock
 
 ```ts
-const app = createRavenflowApp(config, {
-  // Required: Redis connection
-  redis: { host: "localhost", port: 6379, password: "secret", db: 0 },
-
-  // Optional: MongoDB (required for stateful-etl and token-bucket)
-  // String form:
-  mongo: "mongodb://localhost:27017/myapp",
-  // Or object form:
-  mongo: { host: "localhost", port: 27017, username: "admin", password: "secret", database: "myapp" },
-
-  // Optional: job retention policy
-  retention: {
-    completed: 1000, // keep last 1000 completed jobs per queue
-    failed: 5000,    // keep last 5000 failed jobs per queue
-  },
+import { defineJob } from 'ravenflow';
 
-  // Optional: logger config (pino)
-  logger: { level: "info", pretty: true },
+export const cleanupTempFiles = defineJob<void, { removed: number }>({
+  trigger: { cron: '0 3 * * *' },
+  handler: async () => ({ removed: await purgeTempDir() }),
 });
 ```
 
-### Rate Limiting
+One occurrence at a time across the fleet. A worker registers the schedules of its queue's jobs
+when it starts. A clock has nothing to pass, so the payload must be `void` or all-optional: anything
+else does not compile.
 
-Even-spacing rate limiter. `perSecond: 10` means 1 request every 100ms (not a burst of 10 then wait).
+### A spool
 
 ```ts
-steps: {
-  extract: { rateLimit: { perMinute: 60 } },  // 1 per second
-  transform: { rateLimit: { perSecond: 20 } }, // 1 every 50ms
-  load: { rateLimit: { perSecond: 5 } },       // 1 every 200ms
-}
-```
-
-### Step Config (Distributed & Stateful ETL)
+import { defineJob } from 'ravenflow';
 
-Each step can have its own retry strategy, rate limit, and target queue:
-
-```ts
-steps: {
-  transform: {
-    rateLimit: { perSecond: 10 },
-    attempts: 3,
-    backoff: { type: "exponential", delay: 1000 },
-    queue: "heavy-transforms", // route to a different queue
-  },
-}
-```
-
-### Cron Scheduling
-
-Jobs with a `cron` field are automatically scheduled when the worker starts:
-
-```ts
-defineSimpleJob({
-  cron: "*/5 * * * *",  // every 5 minutes
-  handler: async () => { /* ... */ },
+export const enrichDocument = defineJob<PendingDocument, void>({
+  pace: { perSecond: 10 },
+  trigger: { spool: { name: 'pending_documents' } },
+  handler: async ({ params }) => enrich(params),
 });
 ```
 
-## Sending Jobs
-
 ```ts
-// Fire and forget — returns BullMQ Job object
-const job = await app.jobs.sendEmail.send({ to: "user@example.com", subject: "Hi" });
+await client.jobs.enrichDocument.pushMany([
+  { url: 'https://example.com/a', tenantId: 'acme' },
+  { url: 'https://example.com/b', tenantId: 'globex' },
+]);
+```
 
-// Wait for the result
-const result = await app.jobs.sendEmail.sendAndWait({ to: "user@example.com", subject: "Hi" });
+Items wait in MongoDB (`rav:spool-backlog:<name>`). Every 30 seconds, the worker serving the job's
+queue releases at most one tick's work at the job's pace into Redis, and an item leaves the
+backlog once its run ends. Options: `tickMs`, `canPull`, `expireUnclaimedAfterMs`.
 
-// ETL jobs don't take parameters
-await app.jobs.syncUsers.send();
+## Advanced
 
-// ETL sendAndWait returns counts
-const counts = await app.jobs.syncUsers.sendAndWait();
-// { extractedCount: 500, transformedCount: 500, loadedCount: 500 }
-```
-
-## Worker
+**Chains.** A handler that returns `continueWith(nextParams)` ends its link and queues the next one,
+each link a job of its own with its own retries.
 
 ```ts
-// Start processing jobs from a queue
-await app.startWorker("default", { concurrency: 20 });
-
-// Graceful shutdown
-await app.close();
+import { defineJob } from 'ravenflow';
+
+export const syncCatalog = defineJob<{ cursor?: string; synced?: number }, { synced: number }>({
+  trigger: { cron: '0 2 * * *' },
+  handler: async ({ params, continueWith }) => {
+    const page = await api.fetchPage(params.cursor);
+    await db.products.upsertMany(page.items);
+    const synced = (params.synced ?? 0) + page.items.length;
+    if (page.next) return continueWith({ cursor: page.next, synced });
+    return { synced };
+  },
+});
 ```
 
-When `startWorker` is called:
-- Cron schedules are synced to Redis
-- The Stall Watcher starts automatically if stateful-etl jobs are registered
-- The worker begins processing jobs
+**Deduplication.** `deduplicate: true` keeps one run of the job in flight at a time,
+`deduplicate: (params) => id` one per id, and `send(params, { deduplicationId })` chooses for one
+send. A duplicate is ignored, and `send` resolves with the job already holding the id.
 
-## Config Validation
+**Retries.** `attempts: 5, backoff: { type: 'exponential', delayMs: 1_000 }`. A handler runs at
+least once, not exactly once: make it safe to run twice.
 
-Ravenflow validates your config at startup and throws clear errors:
+**Pace.** `pace: { perSecond: 2 }` holds a rate for the whole job, across every worker process.
+A job that comes too early waits its turn without spending an attempt.
 
-```
-[ravenflow] Config validation failed:
-  - "migrateLegacy" (stateful-etl) requires a mongo connection.
-  - "rateLimiter" (token-bucket) requires a cron schedule.
-```
+**Levers.** Every job has `client.jobs.<name>.controls`:
 
-## Companion Packages
+| Lever | On | Effect |
+| --- | --- | --- |
+| `pauseExecution()` / `resumeExecution()` | every job | Jobs are deferred, not lost, and no attempt is spent. |
+| `pauseTrigger()` / `resumeTrigger()` | a clock or a spool | Occurrences are skipped; a spool stops draining. |
+| `pauseIngest()` / `resumeIngest()` | a spool | `push` throws a `SpoolIngestPausedError`. |
+| `purgeBacklog()` | a spool | Drops the items never claimed. |
 
-- [`@ravenflow/mongo`](https://www.npmjs.com/package/@ravenflow/mongo) — Typed MongoDB collections, base repository, Zod schema integration
-- [`@ravenflow/http`](https://www.npmjs.com/package/@ravenflow/http) — HTTP client utilities built on undici
+**Job results.** A worker writes one record per execution to MongoDB (`rav:job-results`): the
+params, the result or the error and its causes, what the handler logged, how long it ran. Kept 14
+days, and counted per minute into `rav:aggregated-results`.
 
-## Worker metrics (pulse / throughput observability)
+**Withdrawal.** A worker that cannot reach MongoDB for 30 seconds closes itself, sets
+`process.exitCode` to 1 and calls `withdrawal.onWithdraw(reason)`. Let your process manager restart
+it.
 
-`createWorker` enables BullMQ's native per-minute metrics collection by default. Every Worker constructed through ravenflow records a monotonic completed-job count + a 10_080-bucket rolling minute histogram under the `bull:<queue>:metrics:completed` Redis keys. The `@ravenflow/dashboard` Jobs-per-Second live indicator reads those keys via `queue.getMetrics('completed')`.
-
-```ts
-// Default — metrics enabled, 7-day window (~10-40 KB per queue in Redis).
-const worker = createWorker(queueName, connection, concurrency, processJob);
-
-// Narrow the window to 1 hour.
-import { METRICS_TIME } from "ravenflow";
-const worker = createWorker(queueName, connection, concurrency, processJob, {
-  metrics: { maxDataPoints: METRICS_TIME.ONE_HOUR },
-});
-
-// Fully opt out — no metrics collection, no Redis footprint.
-const worker = createWorker(queueName, connection, concurrency, processJob, {
-  metrics: undefined,
-});
-```
+## Configuration
 
-See `CHANGELOG.md` for the rationale and Redis cost breakdown.
+| `defineConfig` | Default |
+| --- | --- |
+| `defaultQueue`, `jobs` | required |
+| `logger: { level, pretty }` | `debug` and readable; `info` and JSON with `NODE_ENV=production` |
+| `queuesRetention: { completedCount, failedCount }` | 1000 each |
+| `jobResultsRetentionHours` | 336 |
+| `aggregatedResultsRetentionHours` | 2160 |
+
+| `createRavenflowWorker` | Default |
+| --- | --- |
+| `redis`, `mongo`, `queue` | required |
+| `concurrency` | 5 |
+| `withdrawal: { afterMs, onWithdraw }` | 30 000 ms |
 
 ## License