queasy 0.2.0 → 0.3.0

package/CLAUDE.md

@@ -21,7 +21,7 @@ Tests require a running Redis instance. Use `docker:up` first if needed.
 
 Queasy is a Redis-backed job queue with **at-least-once** delivery semantics. The core logic lives in two layers:
 
-- **JS layer** (`src/queue.js`): The `queue()` factory returns `{ dispatch, cancel, listen }`. On first use, it uploads the Lua script to Redis via `FUNCTION LOAD REPLACE`. A `WeakSet` (`initializedClients`) tracks which Redis clients have already had the functions loaded. `listen()` is currently a TODO stub.
+- **JS layer** (`src/client.js`): The `Client` class accepts a `RedisOptions` object and constructs its own node-redis connection via `createClient` (plain object) or `createCluster` (object with `rootNodes`). On construction it connects, then uploads the Lua script to Redis via `FUNCTION LOAD REPLACE`. The connection is torn down in `close()` via `destroy()`.
 - **Lua layer** (`src/queasy.lua`): All queue state mutations are atomic Redis functions registered under the `queasy` library. No queue logic should be duplicated in JS — the Lua functions are the single source of truth for state transitions.
 
 ### Redis data structures

package/Readme.md

@@ -13,7 +13,7 @@ A Redis-backed job queue for Node.js, featuring (in comparison with design inspi
 
 ### Terminology
 
-A _client_ is an instance of Queasy that connects to a Redis database. A _job_ is the basic unit of work that is _dispatched_ into a _queue_.
+A _client_ is an instance of Queasy. It manages its own Redis connection. A _job_ is the basic unit of work that is _dispatched_ into a _queue_.
 
 A _handler_ is JavaScript code that performs work. There are two kinds of handlers: _task handlers_, which process jobs, and _fail handlers_, which are invoked when a job fails permanently. Handlers run on _workers_, which are Node.js worker threads. By default, a Queasy client automatically creates one worker per CPU.
 
@@ -55,9 +55,14 @@ The response of the heartbeat Lua function indicates whether the client had been
 
 ## API
 
-### `client(redisConnection, workerCount)`
-Returns a Queasy client.
-- `redisConnection`: a node-redis connection object.
+### `new Client(options, workerCount)`
+Returns a Queasy client. Queasy creates and manages its own Redis connection internally.
+- `options`: connection options. Two forms are accepted:
+  - **Single node** (plain object): passed to node-redis `createClient`. Accepts `url`, `socket`, `username`, `password`, and `database`. Defaults to `{}` (connects to `localhost:6379`).
+  - **Cluster** (object with `rootNodes`): passed to node-redis `createCluster`. Accepts:
+    - `rootNodes`: array of per-node connection options (same fields as single-node form); at least three nodes are recommended.
+    - `defaults`: options shared across all nodes (e.g. auth and TLS).
+    - `nodeAddressMap`: address translation map for NAT environments.
 - `workerCount`: number; Size of the worker pool. If 0, or if called in a queasy worker thread, no pool is created. Defaults to the number of CPUs.
 
 The client object returned is an EventEmitter, which emits a 'disconnect' event when it fails permanently for any reason, such as library version mismatch between different workers connected to the same Redis insance, or a lost locks situation. When this happens, in general the application should exit the worker process and allow the supervisor to restart it.

package/docker-compose.yml

@@ -7,8 +7,6 @@ services:
     ports:
       - '6379:6379'
     command: redis-server --save 60 1 --loglevel warning
-    volumes:
-      - redis-data:/data
     healthcheck:
       test: ['CMD', 'redis-cli', 'ping']
       interval: 5s

package/fuzztest/Readme.md

@@ -0,0 +1,185 @@
+# Queasy Fuzz Test Plan
+
+A long-running end-to-end fuzz test that simulates random failures and continuously verifies core system invariants.
+
+## Invariants Verified
+
+1. **Mutual exclusion**: Two jobs with the same Job ID are never processed by different clients or worker threads simultaneously.
+2. **No re-processing of successful jobs**: A job that has succeeded is never processed again.
+3. **Scheduling**: No job is processed before its `run_at` time.
+4. **Priority ordering within a queue**: No job starts processing while another job in the same queue with a lower `run_at` is still waiting (i.e., eligible jobs are dequeued in order).
+5. **Fail handler completeness**: If a fail handler is registered, every job that does not eventually succeed MUST result in the fail handler being invoked.
+6. **Queue progress (priority starvation prevention)**: Non-empty queues at the highest priority level always make progress. When they drain, queues at the next priority level begin making progress.
+
+## Structure Overview
+
+```
+fuzztest/
+  Readme.md              # This file
+  fuzz.js                # Orchestrator: spawns child processes, monitors shared state
+  process.js             # Child process: sets up clients and listens on all queues
+  handlers/
+    periodic.js          # Re-queues itself; dispatches cascade jobs; occasionally stalls/crashes
+    cascade-a.js         # Dispatched by periodic; dispatches into cascade-b
+    cascade-b.js         # Dispatched by cascade-a; final handler
+    fail-handler.js      # Shared fail handler for all queues; records invocations
+  shared/
+    state.js             # In-process shared state helpers (for the orchestrator)
+    log.js               # Structured logger (writes to fuzz-output.log, never throws)
+```
+
+## Process Architecture
+
+The orchestrator (`fuzz.js`) spawns **N child processes** (default: 4). Each child process creates one Redis client and calls `listen()` on every queue. The orchestrator itself does not process jobs — it only monitors invariants and manages the lifecycle.
+
+Handlers write events (job start, finish, fail, stall) directly to a Redis stream (`fuzz:events`). The orchestrator reads from this stream and maintains a shared in-memory log of events, checking invariants after each one. Child processes do not need to forward events to the orchestrator themselves — the stream is the shared channel.
+
+Child processes are deliberately killed and restarted periodically to simulate crashes. A killed process' checked-out jobs will be swept and retried/failed by the remaining processes.
+
+## Queue Configuration
+
+Three queues at different priority levels, all listened on by every child process. Parameters are kept small to produce many events quickly:
+
+| Parameter | `{fuzz}:periodic` | `{fuzz}:cascade-a` | `{fuzz}:cascade-b` |
+|---|---|---|---|
+| Handler | `periodic.js` | `cascade-a.js` | `cascade-b.js` |
+| Priority | 300 | 200 | 100 |
+| `maxRetries` | 3 | 3 | 3 |
+| `maxStalls` | 2 | 2 | 2 |
+| `minBackoff` | 200 ms | 200 ms | 200 ms |
+| `maxBackoff` | 2 000 ms | 2 000 ms | 2 000 ms |
+| `timeout` | 3 000 ms | 3 000 ms | 3 000 ms |
+| `size` | 10 | 10 | 10 |
+| `failHandler` | `fail-handler.js` | `fail-handler.js` | `fail-handler.js` |
+| `failRetryOptions.maxRetries` | 5 | 5 | 5 |
+| `failRetryOptions.minBackoff` | 200 ms | 200 ms | 200 ms |
+
+The short `timeout` (3 s) means stalling jobs are detected and swept quickly. The short `minBackoff` / `maxBackoff` window (200 ms – 2 s) means retries cycle fast. With `maxRetries: 3` and `maxStalls: 2`, most failed jobs reach the fail handler within seconds.
+
+## Periodic Jobs (Seed)
+
+A fixed set of periodic job IDs (e.g., `periodic-0` through `periodic-4`) are dispatched by the orchestrator at startup. Each periodic handler:
+
+1. Records the current processing event by writing `{ type: 'start', queue, id, threadId, clientId, startedAt }` to the `fuzz:events` Redis stream.
+2. Optionally sleeps for a random short delay.
+3. Dispatches a cascade-a job with a unique ID and a `runAt` randomly up to 2 seconds in the future.
+4. Re-dispatches itself (same job ID, `updateRunAt: true`) with a delay of 1–5 seconds, so the job continues to fire periodically.
+5. On success, writes `{ type: 'finish', queue, id, threadId, clientId, finishedAt }` to the `fuzz:events` stream.
+
+The fail handler for periodic jobs also re-dispatches the same periodic job ID (with a delay), ensuring periodic jobs survive permanent failures. This lets the orchestrator assert that periodic jobs keep running indefinitely.
+
+## Cascade Jobs
+
+`cascade-a.js`:
+- Records start/finish events.
+- Dispatches one or two `cascade-b` jobs with unique IDs.
+- Subject to all chaos behaviors (see below).
+
+`cascade-b.js`:
+- Records start/finish events.
+- Terminal handler; does not dispatch further jobs.
+- Subject to all chaos behaviors (see below).
+
+## Chaos Behaviors
+
+All handlers are subject to all chaos behaviors. The probabilities below are per-invocation and apply uniformly across `periodic.js`, `cascade-a.js`, and `cascade-b.js`:
+
+| Behavior | Probability | Notes |
+|---|---|---|
+| Normal completion | ~65% | Dispatches downstream jobs (if any), then returns |
+| Retriable error (throws `Error`) | ~15% | No downstream dispatch |
+| Permanent error (throws `PermanentError`) | ~5% | No downstream dispatch |
+| Stall (returns a never-resolving promise) | ~10% | Detected after `timeout` (3 s); counts as a stall |
+| CPU spin (blocks the worker thread) | ~3% | Tight loop until the process detects the hang and kills the thread (via `timeout`) |
+| Crash (causes the child process to exit) | ~2% | Handler writes a "crash-me" flag to Redis; main thread polls and exits |
+
+With `timeout: 3000`, stalling and spinning jobs are swept within ~3–13 seconds (timeout + heartbeat sweep interval). With `maxStalls: 2`, two stalls exhaust the stall budget and the job is sent to the fail handler, cycling fast.
+
+When a child process crashes, the orchestrator detects the exit event and restarts a new child process after a short delay.
+
+## Event Logging and Invariant Checking
+
+The orchestrator maintains an append-only in-memory event log. Each entry contains:
+```js
+{ type, queue, id, threadId, clientId, timestamp }
+```
+where `type` is one of: `start`, `finish`, `fail`, `stall`, `cancel`.
+
+After each event is appended, the orchestrator runs incremental invariant checks:
+
+### Invariant 1: Mutual Exclusion
+Maintain a `Map<jobId, { clientId, threadId, startedAt }>` of currently-active jobs. On `start`, check that the job ID is not already in the map. On `finish`/`fail`/`stall`, remove it.
+
+If a `start` event arrives for a job ID already in the map → **VIOLATION**.
+
+### Invariant 2: No Re-processing of Succeeded Jobs
+Maintain a `Set<jobId>` of successfully finished job IDs. On `start`, check that the ID is not in this set.
+
+If a `start` event arrives for a job ID in the succeeded set → **VIOLATION**.
+
+Note: Re-processing after a stall or retry is expected and must not be flagged.
+
+### Invariant 3: Scheduling (No Early Processing)
+Each `start` event includes `startedAt` (wall clock). Each job dispatch records an intended `runAt`. On `start`, verify `startedAt >= runAt - CLOCK_TOLERANCE_MS`.
+
+If `startedAt < runAt - CLOCK_TOLERANCE_MS` → **VIOLATION**.
+
+`CLOCK_TOLERANCE_MS` accounts for clock skew between the orchestrator, child processes, and Redis (default: 100ms).
+
+### Invariant 4: Priority Ordering
+Track the earliest-known `runAt` for jobs dispatched into each queue but not yet started. When a `start` event arrives for a job in that queue, verify no other eligible job (with `runAt <= now`) in the same queue has a lower `runAt` that has been waiting longer.
+
+This invariant is best-effort and checked with a configurable lag (e.g., 200ms) to account for the inherent race between dequeue polling and dispatch. A violation is only flagged when the ordering difference exceeds this lag.
+
+### Invariant 5: Fail Handler Completeness
+Track every job that has been dispatched (by ID). When a job exceeds its `maxRetries` or receives a permanent error, a fail event should be observed. Maintain a map `{ jobId → { exhausted: bool, failSeen: bool } }`. After a configurable drain period (e.g., 30 seconds after a queue goes quiet), check that every exhausted job has a corresponding `fail` event.
+
+### Invariant 6: Queue Progress
+The orchestrator monitors the time since the last `start` event per queue. If a queue is known to be non-empty (based on dispatched vs finished counts) and no `start` event has been seen for more than a configurable `STALL_THRESHOLD_MS` (e.g., 30 seconds), flag a progress violation.
+
+Priority starvation is checked by verifying that the low-priority queue does not process jobs while the high-priority queue has outstanding jobs older than the dequeue poll interval.
+
+## Output and Reporting
+
+Violations are logged to stdout and to `fuzz-output.log` with full context. The process does **not** exit on a violation — it logs and continues, accumulating a count of violations. A summary is printed periodically (every 60 seconds) and on `SIGINT`.
+
+Log format (newline-delimited JSON):
+```json
+{ "time": "...", "level": "info|warn|error", "msg": "...", "data": { ... } }
+```
+
+Violation entries use level `"error"` and include the invariant name, the offending event, and relevant recent history.
+
+## Configuration
+
+All tunable parameters live at the top of `fuzz.js` as named constants:
+
+```js
+const NUM_PROCESSES = 4;         // Child processes
+const NUM_PERIODIC_JOBS = 5;     // Fixed periodic job IDs
+const PERIODIC_MIN_DELAY = 1000; // ms before re-queuing self
+const PERIODIC_MAX_DELAY = 5000;
+const CRASH_INTERVAL_MS = 30000; // Orchestrator kills a random child process this often
+const CLOCK_TOLERANCE_MS = 100;
+const STALL_THRESHOLD_MS = 30000;
+const PRIORITY_LAG_MS = 200;
+const LOG_FILE = 'fuzz-output.log';
+```
+
+## Running
+
+The fuzz test is separate from the default test suite and is never run by `npm test`. It is started manually:
+
+```sh
+node fuzztest/fuzz.js
+```
+
+It runs indefinitely. Stop with `Ctrl+C`. A summary of violations and events processed will be printed on exit.
+
+## Notes on Implementation
+
+- Child processes use the `queasy` library's public API (`queue()`, `dispatch()`, `listen()`). They do not talk directly to Redis.
+- The orchestrator does not import from `src/`; it only spawns child processes and learns about child process lifecycle only from the `spawn` and `exit` events.
+- All handler modules in `fuzztest/handlers/` must be self-contained ESM modules that can be passed as `handlerPath` to `queue.listen()`.
+- Handlers write events to the `fuzz:events` Redis stream using a dedicated Redis client created at handler module load time. The orchestrator reads from this stream via `XREAD BLOCK`. This is the only communication channel between handlers and the orchestrator — no IPC is used.
+- The chaos crash behavior must be triggered from the child process's main thread, not from inside a handler's worker thread. To simulate a crash, the handler uses `postMessage` to send a `{ type: 'crash' }` message to the main thread, which listens for it and calls `process.exit()`.

package/fuzztest/fuzz.js

@@ -0,0 +1,354 @@
+/**
+ * Fuzz test orchestrator.
+ *
+ * - Spawns NUM_PROCESSES child processes, each running fuzztest/process.js
+ * - Dispatches seed periodic jobs at startup
+ * - Reads events from the fuzz:events Redis stream
+ * - Checks system invariants after each event
+ * - Logs violations without terminating
+ * - Prints a summary every 60 seconds and on SIGINT
+ */
+
+import { fork } from 'node:child_process';
+import { createWriteStream } from 'node:fs';
+import { dirname, join } from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { createClient } from 'redis';
+import { Client } from '../src/index.js';
+import { readEvents, STREAM_KEY } from './shared/stream.js';
+
+const __dirname = dirname(fileURLToPath(import.meta.url));
+
+// ── Configuration ──────────────────────────────────────────────────────────────
+
+const NUM_PROCESSES = 4;
+const NUM_PERIODIC_JOBS = 5;
+const CRASH_INTERVAL_MS = 30_000; // kill a random process this often
+const CLOCK_TOLERANCE_MS = 200; // allow this much early-start slop
+const STALL_THRESHOLD_MS = 30_000; // no start event → progress violation
+const PRIORITY_LAG_MS = 500; // ordering slop between queues
+const PROCESS_RESTART_DELAY_MS = 500;
+const SUMMARY_INTERVAL_MS = 60_000;
+const LOG_FILE = join(__dirname, '..', 'fuzz-output.log');
+
+// ── Logging ────────────────────────────────────────────────────────────────────
+
+const logStream = createWriteStream(LOG_FILE, { flags: 'a' });
+
+function log(level, msg, data = {}) {
+    const entry = JSON.stringify({ time: new Date().toISOString(), level, msg, data });
+    if (level === 'error') process.stdout.write(`VIOLATION: ${entry}\n`);
+    else process.stdout.write(`${entry}\n`);
+    logStream.write(`${entry}\n`);
+}
+
+// ── Invariant state ────────────────────────────────────────────────────────────
+
+/** @type {Map<string, {queue: string, startedAt: number, runAt: number, pid: number}>} */
+const activeJobs = new Map();
+
+/** @type {Set<string>} */
+const succeededJobs = new Set();
+
+/**
+ * Per-queue: list of {id, runAt, dispatchedAt} for jobs seen but not yet started.
+ * Used to check priority ordering.
+ * @type {Map<string, {id: string, runAt: number, dispatchedAt: number}[]>}
+ */
+const waitingByQueue = new Map();
+
+/** @type {Map<string, number>} last start event timestamp per queue */
+const lastStartPerQueue = new Map();
+
+let violationCount = 0;
+let eventCount = 0;
+
+function violation(invariant, msg, data = {}) {
+    violationCount++;
+    log('error', `[${invariant}] ${msg}`, data);
+}
+
+// ── Invariant checks ───────────────────────────────────────────────────────────
+
+/**
+ * Called when a child process dequeues a job (via IPC).
+ * @param {number} pid
+ * @param {{queue: string, jobId: string, runAt: number}} msg
+ */
+function onIpcDequeue(pid, msg) {
+    const { jobId: id, queue, runAt } = msg;
+
+    // Mutual exclusion: job must not already be active
+    if (activeJobs.has(id)) {
+        const existing = activeJobs.get(id);
+        violation('MutualExclusion', `Job ${id} dequeued while already active`, {
+            existing,
+            newDequeue: { queue, pid },
+        });
+    }
+
+    activeJobs.set(id, { queue, pid, startedAt: Date.now(), runAt });
+}
+
+/**
+ * Called when a child process finishes/retries/fails a job (via IPC).
+ * @param {string} jobId
+ */
+function onIpcJobDone(jobId) {
+    activeJobs.delete(jobId);
+}
+
+function onStart(event) {
+    const { id, queue, runAt: runAtStr, startedAt: startedAtStr } = event;
+    const runAt = Number(runAtStr);
+    const startedAt = Number(startedAtStr);
+
+    // No re-processing of succeeded jobs (except periodic which re-queues itself)
+    if (succeededJobs.has(id) && !id.startsWith('fuzz-periodic-')) {
+        violation('NoReprocess', `Job ${id} started after already succeeding`, {
+            queue,
+            startedAt,
+        });
+    }
+
+    // Scheduling: not before runAt
+    if (runAt > 0 && startedAt < runAt - CLOCK_TOLERANCE_MS) {
+        violation('Scheduling', `Job ${id} started ${runAt - startedAt}ms too early`, {
+            queue,
+            runAt,
+            startedAt,
+            delta: runAt - startedAt,
+        });
+    }
+
+    // Priority ordering: no eligible lower-runAt job in same queue waiting
+    const waiting = waitingByQueue.get(queue) ?? [];
+    for (const w of waiting) {
+        if (w.id === id) continue;
+        if (w.runAt <= startedAt - CLOCK_TOLERANCE_MS) {
+            if (runAt > w.runAt + PRIORITY_LAG_MS) {
+                violation(
+                    'Ordering',
+                    `Job ${id} (runAt=${runAt}) started before ${w.id} (runAt=${w.runAt}) in ${queue}`,
+                    {
+                        startedId: id,
+                        startedRunAt: runAt,
+                        waitingId: w.id,
+                        waitingRunAt: w.runAt,
+                    }
+                );
+                break;
+            }
+        }
+    }
+
+    lastStartPerQueue.set(queue, startedAt);
+
+    // Remove from waiting list
+    if (waitingByQueue.has(queue)) {
+        waitingByQueue.set(
+            queue,
+            waitingByQueue.get(queue).filter((w) => w.id !== id)
+        );
+    }
+}
+
+function onFinish(event) {
+    succeededJobs.add(event.id);
+}
+
+/**
+ * Called when a child process exits. Clears all active jobs belonging to that
+ * PID so they don't trigger spurious MutualExclusion violations when the
+ * queasy sweep retries them and a new process picks them up.
+ * @param {number} pid
+ */
+function onProcessExit(pid) {
+    for (const [id, entry] of activeJobs) {
+        if (entry.pid === pid) {
+            activeJobs.delete(id);
+        }
+    }
+}
+
+/**
+ * Called periodically to check queue progress and priority starvation.
+ */
+function checkProgress() {
+    const now = Date.now();
+    for (const [queue, lastStart] of lastStartPerQueue) {
+        const idle = now - lastStart;
+        if (idle > STALL_THRESHOLD_MS) {
+            violation('Progress', `Queue ${queue} has not processed a job in ${idle}ms`, {
+                queue,
+                lastStartAt: lastStart,
+                idleMs: idle,
+            });
+        }
+    }
+
+    // Priority starvation: cascade-b should not start while periodic has old eligible jobs
+    const periodicWaiting = waitingByQueue.get('{fuzz}:periodic') ?? [];
+    const eligiblePeriodic = periodicWaiting.filter((w) => w.runAt <= now - PRIORITY_LAG_MS);
+    if (eligiblePeriodic.length > 0) {
+        const lastBStart = lastStartPerQueue.get('{fuzz}:cascade-b') ?? 0;
+        const bStartedAfterPeriodic = eligiblePeriodic.some(
+            (w) => lastBStart > w.dispatchedAt + PRIORITY_LAG_MS
+        );
+        if (bStartedAfterPeriodic) {
+            violation(
+                'PriorityStarvation',
+                'cascade-b processed while periodic had eligible waiting jobs',
+                {
+                    eligiblePeriodicCount: eligiblePeriodic.length,
+                }
+            );
+        }
+    }
+}
+
+// ── Event dispatch ─────────────────────────────────────────────────────────────
+
+function handleEvent(event) {
+    eventCount++;
+    const { type } = event;
+    log('info', 'event', event);
+
+    if (type === 'start') {
+        const { id, queue, runAt } = event;
+        // Register in waiting list for ordering checks (before onStart removes it)
+        const runAtNum = Number(runAt);
+        const q = waitingByQueue.get(queue) ?? [];
+        if (!q.find((w) => w.id === id)) {
+            q.push({ id, runAt: runAtNum, dispatchedAt: Date.now() });
+            waitingByQueue.set(queue, q);
+        }
+        onStart(event);
+    } else if (type === 'finish') {
+        onFinish(event);
+    }
+}
+
+// ── Summary ────────────────────────────────────────────────────────────────────
+
+function printSummary() {
+    const summary = {
+        events: eventCount,
+        violations: violationCount,
+        activeJobs: activeJobs.size,
+        succeededJobs: succeededJobs.size,
+        lastStartPerQueue: Object.fromEntries(lastStartPerQueue),
+    };
+    log('info', 'Summary', summary);
+    console.log(`\n=== Fuzz Summary ===`);
+    console.log(`  Events processed : ${eventCount}`);
+    console.log(`  Violations found : ${violationCount}`);
+    console.log(`  Active jobs      : ${activeJobs.size}`);
+    console.log(`  Succeeded jobs   : ${succeededJobs.size}`);
+    console.log('===================\n');
+}
+
+// ── Child process management ───────────────────────────────────────────────────
+
+/** @type {Set<import('node:child_process').ChildProcess>} */
+const processes = new Set();
+
+function spawnProcess() {
+    const child = fork(join(__dirname, 'process.js'));
+    processes.add(child);
+
+    child.on('message', (msg) => {
+        if (msg.type === 'dequeue') {
+            onIpcDequeue(child.pid, msg);
+        } else if (msg.type === 'finish' || msg.type === 'retry' || msg.type === 'fail') {
+            onIpcJobDone(msg.jobId);
+        }
+    });
+
+    child.on('exit', (code, signal) => {
+        processes.delete(child);
+        log('info', 'Child process exited', { pid: child.pid, code, signal });
+        onProcessExit(child.pid);
+        setTimeout(spawnProcess, PROCESS_RESTART_DELAY_MS);
+    });
+
+    child.on('error', (err) => {
+        log('info', 'Child process error', { pid: child.pid, error: err.message });
+    });
+
+    return child;
+}
+
+function killRandomProcess() {
+    const list = [...processes];
+    if (list.length === 0) return;
+    const target = list[Math.floor(Math.random() * list.length)];
+    log('info', 'Killing random child process', { pid: target.pid });
+    target.kill('SIGKILL');
+}
+
+// ── Redis setup ────────────────────────────────────────────────────────────────
+
+const redis = createClient();
+const dispatchRedis = createClient();
+
+await redis.connect();
+await dispatchRedis.connect();
+
+// Clean up state from previous runs
+await redis.del(STREAM_KEY);
+log('info', 'Cleared fuzz:events stream from previous run');
+
+// Dispatch seed periodic jobs (await ready to avoid Function not found race)
+const dispatchClient = await new Promise((resolve) => new Client(dispatchRedis, 0, resolve));
+const periodicQueue = dispatchClient.queue('{fuzz}:periodic', true);
+
+for (let i = 0; i < NUM_PERIODIC_JOBS; i++) {
+    const id = `fuzz-periodic-${i}`;
+    await periodicQueue.dispatch({ periodic: true, index: i }, { id });
+    log('info', `Dispatched seed job ${id}`);
+}
+
+// ── Spawn child processes ──────────────────────────────────────────────────────
+
+for (let i = 0; i < NUM_PROCESSES; i++) {
+    spawnProcess();
+}
+
+// Periodically kill a random process to simulate crashes
+const crashTimer = setInterval(killRandomProcess, CRASH_INTERVAL_MS);
+
+// Periodic progress + starvation check
+const progressTimer = setInterval(checkProgress, 10_000);
+
+// Summary timer
+const summaryTimer = setInterval(printSummary, SUMMARY_INTERVAL_MS);
+
+log('info', 'Orchestrator started', {
+    numProcesses: NUM_PROCESSES,
+    numPeriodicJobs: NUM_PERIODIC_JOBS,
+    logFile: LOG_FILE,
+});
+
+// ── SIGINT handler ─────────────────────────────────────────────────────────────
+
+process.on('SIGINT', () => {
+    clearInterval(crashTimer);
+    clearInterval(progressTimer);
+    clearInterval(summaryTimer);
+
+    for (const child of processes) {
+        child.kill('SIGKILL');
+    }
+
+    printSummary();
+    process.exit(violationCount > 0 ? 1 : 0);
+});
+
+// ── Event loop ─────────────────────────────────────────────────────────────────
+
+// Read from the beginning. We cleared the stream above, so '0' reads all
+// events from the fresh start without missing anything.
+for await (const event of readEvents(redis, '0')) {
+    handleEvent(event);
+}

package/fuzztest/handlers/cascade-a.js

@@ -0,0 +1,94 @@
+/**
+ * cascade-a handler — dispatches one or two cascade-b jobs on normal completion.
+ * Subject to all chaos behaviors.
+ */
+
+import { BroadcastChannel } from 'node:worker_threads';
+import { createClient } from 'redis';
+import { Client, PermanentError } from '../../src/index.js';
+import { pickChaos } from '../shared/chaos.js';
+import { emitEvent } from '../shared/stream.js';
+
+const redis = createClient();
+const eventRedis = createClient();
+
+await redis.connect();
+await eventRedis.connect();
+
+// Dispatch-only queasy client (await ready to avoid Function not found race)
+const client = await new Promise((resolve) => new Client(redis, 0, resolve));
+const cascadeBQueue = client.queue('{fuzz}:cascade-b', true);
+
+const crashChannel = new BroadcastChannel('fuzz-crash');
+
+/**
+ * @param {any} data
+ * @param {import('../../src/types.js').Job} job
+ */
+export async function handle(_data, job) {
+    const startedAt = Date.now();
+    await emitEvent(eventRedis, {
+        type: 'start',
+        queue: '{fuzz}:cascade-a',
+        id: job.id,
+        pid: String(process.pid),
+        runAt: String(job.runAt),
+        startedAt: String(startedAt),
+    });
+
+    const chaos = pickChaos();
+    await emitEvent(eventRedis, {
+        type: 'chaos',
+        queue: '{fuzz}:cascade-a',
+        id: job.id,
+        chaos,
+    });
+
+    if (chaos === 'crash') {
+        crashChannel.postMessage({ type: 'crash' });
+        await new Promise(() => {});
+    }
+
+    if (chaos === 'stall') {
+        await new Promise(() => {});
+    }
+
+    if (chaos === 'spin') {
+        const end = Date.now() + 10_000;
+        while (Date.now() < end) {
+            /* busy wait */
+        }
+    }
+
+    if (chaos === 'permanent') {
+        throw new PermanentError('cascade-a: permanent chaos');
+    }
+
+    if (chaos === 'retriable') {
+        throw new Error('cascade-a: retriable chaos');
+    }
+
+    // Normal completion: dispatch 1-2 cascade-b jobs
+    const count = Math.random() < 0.5 ? 1 : 2;
+    const runAtOffset = Math.random() * 2000;
+    const dispatchPromises = [];
+    for (let i = 0; i < count; i++) {
+        dispatchPromises.push(
+            cascadeBQueue.dispatch(
+                { from: job.id, index: i },
+                {
+                    runAt: Date.now() + runAtOffset,
+                }
+            )
+        );
+    }
+    const ids = await Promise.all(dispatchPromises);
+
+    await emitEvent(eventRedis, {
+        type: 'finish',
+        queue: '{fuzz}:cascade-a',
+        id: job.id,
+        finishedAt: String(Date.now()),
+        dispatched: ids.join(','),
+    });
+}