glide-mq 0.8.0 → 0.9.0

package/CHANGELOG.md

@@ -6,7 +6,72 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ---
 
-## [Unreleased]
+## [0.9.0] - 2026-03-08
+
+### Added
+
+- Subject-based filtering for `BroadcastWorker` - NATS-style wildcard matching on job names. Configure via `subjects` option with `*` (single-token) and `>` (multi-token) wildcards. Non-matching messages are auto-acknowledged. Exported `matchSubject` and `compileSubjectMatcher` utilities (#119).
+- `Producer` class - lightweight job enqueuing for serverless/edge environments without EventEmitter or Job instances. Returns plain string IDs. Use with `ServerlessPool` for automatic connection reuse across warm Lambda/Edge invocations. API: `add(name, data, opts)`, `addBulk(jobs)`, `close()` (#112).
+- `ServerlessPool` and `serverlessPool` singleton - connection pooling for serverless environments. Caches Producer instances by queue name and connection fingerprint. API: `getProducer(name, opts)`, `closeAll()`, `size` (#112).
+- LIFO (Last-In-First-Out) job processing via `lifo: true` option. Uses dedicated Valkey LIST with RPUSH/RPOP. Priority and delayed jobs take precedence. Cannot be combined with ordering keys (#87).
+- Time-series metrics - `queue.getMetrics(type, opts?)` returns per-minute throughput and latency data with 24-hour retention. Zero extra RTTs (#82).
+- `opts.jobId` - custom job IDs for deterministic identity. Max 256 characters (#79).
+- `queue.addAndWait(name, data, { waitTimeout })` - enqueue and wait for completion without polling.
+- `job.moveToDelayed(timestampMs, nextStep?)` - pause active job mid-processor for step-job workflows.
+- `DelayedError` - exported error type for step-job control.
+- Batch processing via `batch: { size, timeout? }` option. Processor receives `Job[]`, returns `R[]`. `BatchError` for partial failure (#81).
+- `glide-mq/proxy` subpath - HTTP proxy for cross-language job enqueue. REST endpoints with queue allowlist, 1MB limit, graceful shutdown (#83).
+- Wire protocol documentation (`docs/WIRE_PROTOCOL.md`) - raw FCALL reference for any language (#83).
+- DAG workflows - `FlowProducer.addDAG()` and `dag()` helper for arbitrary DAG topologies (#86).
+- Serverless usage guide (`docs/SERVERLESS.md`) - Lambda, Cloudflare Workers, Vercel Edge examples.
+- List-active counter self-healing via `glidemq_healListActive` Lua function. Automatically corrects counter drift caused by worker crashes during scheduler promotion ticks (#124).
+- Proxy endpoints: `GET /queues/:name/jobs` (list/filter), `DELETE /queues/:name/jobs/:id` (#124).
+- CI: `npm audit` security scanning, `timeout-minutes` on all jobs, `npm ci` with cache in publish workflow (#124).
+
+### Fixed
+
+- 62 issues from deep project audit across 7 domains (security, performance, code quality, architecture, testing, backend, devops) (#124):
+  - **Critical**: Worker heartbeat unhandled rejections, proxy validation gaps (NaN/Infinity), proxy queue cache race condition, poll loop promise handling on close, cross-queue parent registration error handling.
+  - **Security**: Sandbox path traversal protection via `realpathSync`, proxy input validation with `Number.isFinite`, queue name length limit (256 chars).
+  - **Performance**: Lua metrics HKEYS scan frequency reduced 10x, token bucket early exit, DAG string parsing O(n) to O(1).
+  - **Reliability**: Worker/Producer `close()` with double-close guard and closed flag, `QueueEvents` recursive poll guard, sandbox pool exit/error listener cleanup, serverless pool closing state guard.
+  - **Proxy**: Configurable `onError` callback (replaces silent error swallowing), graceful shutdown with draining flag, pause/resume returns 200 with state.
+- `globalConcurrency` enforced for LIFO/priority-list jobs via atomic `rpopAndReserve` (#87).
+- Scheduler LIFO forwarding and FlowProducer child LIFO routing (#87).
+- `list-active` counter DECR on job removal/deferral (#87).
+- Function library bumped to version 60.
+
+### Changed
+
+- **Breaking (internal)**: `Worker` and `BroadcastWorker` now extend `BaseWorker` abstract class. Public API unchanged. Eliminates ~1400 lines of duplication (3407 to 2024 lines, 41% reduction) (#124).
+- Worker uses explicit state machine (7 states: created, initializing, running, paused, draining, closing, closed) replacing boolean flags (#124).
+- Proxy pause/resume endpoints return 200 with `{ paused: boolean }` instead of 204 (#124).
+- Proxy health endpoint includes `queues` count (#124).
+- Test suite: 24 hardcoded `setTimeout` waits replaced with `waitFor` predicates (#124).
+- 79 eslint `no-unused-vars` warnings resolved across test files (#124).
+
+---
+
+## [0.8.1] - 2026-02-27
+
+### Security
+
+- Reject invalid cron patterns: zero step (`*/0`), out-of-bounds values, reversed ranges, malformed tokens (#56).
+- Enforce 1MB payload limit on job data, progress, and logs using `Buffer.byteLength` for correct UTF-8 byte counting. Covers `add`, `addBulk`, `updateData`, `updateProgress`, and `log` (#61).
+- Fix path leak in sandbox error messages (#54).
+
+### Performance
+
+- Hierarchical cron search replacing brute-force minute iteration - 4400x speedup for yearly schedules. UTC-correct date handling, 10-year search horizon (#59).
+- Batch Redis commands in `Job.retry()` and `updateProgress()` (#53).
+
+### Added
+
+- Comprehensive local fuzzer with pre-push hook.
+
+### Docs
+
+- Dashboard section in README, feature map improvements (#57, #58).
 
 ---
 

package/README.md

@@ -1,58 +1,42 @@
 # glide-mq
 
-**High-performance message queue for Node.js** — powered by Valkey/Redis Streams and a Rust-native NAPI client.
+[![npm version](https://img.shields.io/npm/v/glide-mq)](https://www.npmjs.com/package/glide-mq)
+[![license](https://img.shields.io/npm/l/glide-mq)](https://github.com/avifenesh/glide-mq/blob/main/LICENSE)
+[![CI](https://github.com/avifenesh/glide-mq/actions/workflows/ci.yml/badge.svg)](https://github.com/avifenesh/glide-mq/actions/workflows/ci.yml)
+[![node](https://img.shields.io/node/v/glide-mq)](https://nodejs.org/)
 
-If you find this useful, [give it a ⭐ on GitHub](https://github.com/avifenesh/glide-mq) — it helps the project reach more developers.
+High-performance message queue for Node.js built on Valkey/Redis Streams with 1-RTT job operations and cluster-native design.
 
-```bash
-npm install glide-mq
-```
+glide-mq is for anyone building background jobs, task queues, or workflow orchestration in Node.js. It connects through a Rust-native NAPI client ([valkey-glide](https://github.com/valkey-io/valkey-glide)), executes all queue logic in a single Valkey Server Function call per operation (FCALL, not EVAL), and hash-tags every key for automatic cluster slot alignment. The result is fewer round trips, no Lua cache misses, and zero cluster configuration.
+
+> If glide-mq is useful to you, consider giving it a star on [GitHub](https://github.com/avifenesh/glide-mq). It helps others discover the project.
 
 ## Why glide-mq
 
-- **1 RTT per job** — `completeAndFetchNext` finishes the current job and fetches the next one in a single round-trip
-- **Rust core, not ioredis** — built on [Valkey GLIDE](https://github.com/valkey-io/valkey-glide)'s native NAPI bindings for lower latency and less GC pressure
-- **1 function library, not 53 scripts** — all queue logic runs as a single Valkey Server Function (no EVAL overhead)
-- **Cluster-native** — hash-tagged keys work out of the box; no manual `{braces}` needed
-- **Cloud-ready** — AZ-affinity routing and IAM auth built in
+- Use this when you need **throughput**: 48k jobs/s at concurrency=50, 4x faster than BullMQ on the same hardware.
+- Use this when you run **Valkey/Redis clusters**: all keys hash-tagged out of the box, no `{braces}` workarounds.
+- Use this when you need **workflows**: parent-child trees, DAGs with fan-in, step jobs, batch processing, and cron scheduling in one library.
+- Use this when you deploy to **serverless**: lightweight `Producer` and `ServerlessPool` cache connections across warm invocations.
+- Use this when you want **pub/sub with durability**: `Broadcast` delivers to all subscribers with retries, backpressure, and NATS-style subject filtering.
 
-## Features
+## Install
 
-- **Queues & Workers** — producer/consumer with configurable concurrency
-- **Delayed & priority jobs** — schedule jobs for later or run high-priority work first
-- **Workflows** — `FlowProducer` parent-child trees, `chain`, `group`, `chord` pipelines with result aggregation
-- **Schedulers** — cron and interval repeatable jobs, persisted across restarts
-- **Per-key ordering** — sequential processing per key while staying parallel across keys
-- **Rate limiting** — token-bucket (cost-based), per-group, and global rate limiting
-- **Retries & DLQ** — exponential/fixed/custom backoff with dead-letter queues
-- **Deduplication** — simple, throttle, and debounce modes with configurable TTL
-- **Job revocation** — cooperative cancellation via AbortSignal for active jobs
-- **Stalled job recovery** — auto-reclaim jobs from crashed workers via XAUTOCLAIM
-- **Global concurrency** — cross-worker active job cap for the entire queue
-- **Pause & resume** — pause/resume at queue level or per-worker, with force option
-- **Drain** — `queue.drain(delayed?)` removes all waiting (and optionally delayed) jobs in a single server-side call
-- **Real-time events** — `QueueEvents` stream for added, completed, failed, stalled, revoked, and more
-- **Job search** — query by state, name, and data filters
-- **Progress tracking** — real-time numeric or object progress updates
-- **Batch API** — `addBulk` for high-throughput ingestion (12.7× faster than serial)
-- **Compression** — transparent gzip (up to 98% size reduction)
-- **Graceful shutdown** — one-liner `gracefulShutdown()` for SIGTERM/SIGINT handling
-- **Connection sharing** — reuse a single client across components to reduce TCP connections
-- **Observability** — OpenTelemetry tracing, per-job logs, [`@glidemq/dashboard`](https://github.com/avifenesh/glidemq-dashboard) web UI
-- **In-memory testing** — `TestQueue` & `TestWorker` with zero dependencies via `glide-mq/testing`
-
-## Quick Start
+```bash
+npm install glide-mq
+```
+
+Requires Node.js 20+ and a running [Valkey](https://valkey.io) 7.0+ or Redis 7.0+ instance.
+
+## Quick start
 
 ```typescript
 import { Queue, Worker } from 'glide-mq';
 
 const connection = { addresses: [{ host: 'localhost', port: 6379 }] };
 
-// Producer
 const queue = new Queue('tasks', { connection });
 await queue.add('send-email', { to: 'user@example.com', subject: 'Hello' });
 
-// Consumer
 const worker = new Worker('tasks', async (job) => {
   console.log(`Processing ${job.name}:`, job.data);
   return { sent: true };
@@ -62,8 +46,6 @@ worker.on('completed', (job) => console.log(`Job ${job.id} done`));
 worker.on('failed', (job, err) => console.error(`Job ${job.id} failed:`, err.message));
 ```
 
-Requires Node.js 20+ and a running [Valkey](https://valkey.io) (7.0+) or Redis 7.0+ instance.
-
 ## Benchmarks
 
 | Concurrency | Throughput |
@@ -73,28 +55,178 @@ Requires Node.js 20+ and a running [Valkey](https://valkey.io) (7.0+) or Redis 7
 | c=10 | 15,504 jobs/s |
 | c=50 | 48,077 jobs/s |
 
-`addBulk` batch API: **1,000 jobs in 18 ms** (12.7× faster than serial).
+`addBulk` batch API: **1,000 jobs in 18 ms** (12.7x faster than serial).
 Gzip compression: **98% payload reduction** on 15 KB payloads.
 
 *Valkey 8.0, single node, no-op processor. Run `npm run bench` to reproduce.*
 
+## Comparison
+
+| | glide-mq | BullMQ | Bee Queue |
+|---|---|---|---|
+| **Network per job** | 1 RTT (`completeAndFetchNext`) | 4-7 RTTs (lock + complete + fetch) | 2-3 RTTs |
+| **Client** | Rust NAPI ([valkey-glide](https://github.com/valkey-io/valkey-glide)) | ioredis (pure JS) | node_redis (pure JS) |
+| **Server logic** | 1 Valkey Function library (persistent, named) | 53 EVAL scripts (cache-miss prone) | Lua scripts |
+| **Cluster** | Hash-tagged keys, zero config | Manual `{braces}` or workarounds | Not supported |
+| **Workflows** | FlowProducer trees, DAG, chain/group/chord | FlowProducer trees | Not supported |
+| **Pub/sub** | Native Broadcast with subject filtering | Not supported | Not supported |
+| **Serverless** | Producer + ServerlessPool | Not supported | Not supported |
+| **Throughput** | 48k jobs/s (c=50) | ~12k jobs/s (c=50) | ~5k jobs/s (c=50) |
+
+## Core concepts
+
+- **Queue** -- stores jobs in Valkey Streams. Handles enqueue, delay, priority, pause, drain, and bulk operations.
+- **Worker** -- processes jobs with configurable concurrency, prefetch, lock duration, and stalled-job recovery.
+- **Job** -- a unit of work with name, data, options (retries, backoff, priority, TTL), and lifecycle events.
+- **FlowProducer** -- creates parent-child job trees and DAGs. A parent waits for all children before processing.
+- **Producer** -- lightweight enqueue-only client. No EventEmitter, no Job instances, returns plain string IDs. Built for serverless.
+- **Broadcast** -- fan-out pub/sub. Each message is delivered to every subscriber group with independent retries and backpressure.
+- **QueueEvents** -- real-time stream of job lifecycle events (completed, failed, delayed, waiting, etc.).
+
+## Features
+
+### Core
+
+- **Queues and workers** with configurable concurrency, prefetch, and lock duration ([Usage](docs/USAGE.md))
+- **Delayed, priority, and bulk enqueue** for scheduling and high-throughput ingestion ([Usage](docs/USAGE.md))
+- **Batch processing** -- process multiple jobs at once via `batch: { size, timeout? }` ([Usage](docs/USAGE.md#batch-processing))
+- **Request-reply** -- `queue.addAndWait(name, data, { waitTimeout })` for synchronous RPC ([Usage](docs/USAGE.md#request-reply-with-addandwait))
+- **LIFO mode** -- `lifo: true` processes newest jobs first ([Advanced](docs/ADVANCED.md#lifo-mode))
+- **Job TTL** -- auto-expire jobs after a time-to-live window ([Advanced](docs/ADVANCED.md#job-ttl))
+- **Custom job IDs** -- deterministic, idempotent enqueue; duplicates return `null` ([Advanced](docs/ADVANCED.md#custom-job-ids))
+- **Pluggable serializers** -- swap JSON for any `{ serialize, deserialize }` implementation ([Advanced](docs/ADVANCED.md#pluggable-serializers))
+- **Transparent compression** -- gzip payloads at the queue level ([Advanced](docs/ADVANCED.md#transparent-compression))
+
+### Reliability
+
+- **Retries with exponential, fixed, or custom backoff** and dead-letter queues ([Advanced](docs/ADVANCED.md#retries-and-backoff))
+- **UnrecoverableError** -- skip all retries and fail permanently ([Usage](docs/USAGE.md#unrecoverableerror))
+- **Stalled recovery** -- auto-reclaim stuck jobs via consumer group PEL and `XAUTOCLAIM` ([Usage](docs/USAGE.md#worker))
+- **Job revocation** -- cooperative cancellation with `AbortSignal` ([Advanced](docs/ADVANCED.md#job-revocation))
+- **Deduplication** -- simple, throttle, and debounce modes with configurable TTL ([Advanced](docs/ADVANCED.md#deduplication))
+- **Per-key ordering** -- sequential processing per ordering key with configurable group concurrency ([Advanced](docs/ADVANCED.md#ordering-and-group-concurrency))
+- **Rate limiting** -- per-group sliding window, token bucket, and global queue-wide limits ([Advanced](docs/ADVANCED.md#global-rate-limiting))
+- **Sandboxed processors** -- run processors in worker threads or child processes ([Architecture](docs/ARCHITECTURE.md))
+
+### Orchestration
+
+- **FlowProducer** -- parent-child job trees with `chain`, `group`, and `chord` helpers ([Workflows](docs/WORKFLOWS.md))
+- **DAG workflows** -- arbitrary dependency graphs with `FlowProducer.addDAG()` and `dag()` helper; multi-parent fan-in, diamond patterns, cycle detection ([Workflows](docs/WORKFLOWS.md))
+- **Step jobs** -- `job.moveToDelayed(timestamp, nextStep)` suspends a job mid-processor and resumes later ([Usage](docs/USAGE.md#pause-and-resume-a-job-later-step-jobs))
+- **Dynamic children** -- `job.moveToWaitingChildren()` pauses a parent to add children mid-execution ([Workflows](docs/WORKFLOWS.md))
+- **Batch processing** -- process multiple jobs at once for bulk I/O ([Usage](docs/USAGE.md#batch-processing))
+
+### Scheduling
+
+- **Cron and interval schedulers** -- 5-field cron with timezone, fixed intervals, and `repeatAfterComplete` mode ([Advanced](docs/ADVANCED.md#job-schedulers))
+- **Bounded schedulers** -- `limit`, `startDate`, and `endDate` for finite schedules ([Advanced](docs/ADVANCED.md#bounded-schedulers))
+
+### Pub/Sub
+
+- **Broadcast** -- fan-out delivery to all subscriber groups ([Usage](docs/USAGE.md#broadcast--broadcastworker))
+- **BroadcastWorker** -- independent consumer groups with own retries, concurrency, and backpressure ([Usage](docs/USAGE.md#broadcast--broadcastworker))
+- **Subject filtering** -- NATS-style patterns (`*` one segment, `>` trailing wildcard) for topic-based routing ([Usage](docs/USAGE.md#broadcast--broadcastworker))
+
+### Serverless
+
+- **Producer** -- enqueue without EventEmitter overhead, returns plain string IDs ([Usage](docs/USAGE.md))
+- **ServerlessPool** -- connection caching across warm Lambda/Edge invocations ([Serverless](docs/SERVERLESS.md))
+
+### Observability
+
+- **QueueEvents** -- real-time stream-based lifecycle events ([Observability](docs/OBSERVABILITY.md))
+- **Time-series metrics** -- per-minute throughput and latency retained 24h, recorded server-side ([Observability](docs/OBSERVABILITY.md))
+- **OpenTelemetry** -- automatic span emission; bring your own tracer or auto-detect `@opentelemetry/api` ([Observability](docs/OBSERVABILITY.md))
+- **Job logs** -- append structured log entries per job with pagination ([Observability](docs/OBSERVABILITY.md))
+- **Job mutations** -- `changePriority()`, `changeDelay()`, `promote()` after enqueue; `retryJobs()` and `clean()` in bulk ([Usage](docs/USAGE.md))
+- **Graceful shutdown** -- `gracefulShutdown()` helper registers SIGTERM/SIGINT handlers ([Usage](docs/USAGE.md#graceful-shutdown))
+- **In-memory testing** -- `TestQueue` and `TestWorker` with zero Valkey dependency ([Testing](docs/TESTING.md))
+
+### Cloud
+
+- **Cluster-native** -- hash-tagged keys `glide:{queueName}:*` route all queue data to the same slot ([Usage](docs/USAGE.md#cluster-mode))
+- **IAM authentication** -- native SigV4 auth for AWS ElastiCache and MemoryDB ([Usage](docs/USAGE.md#cluster-mode))
+- **AZ-affinity routing** -- `readFrom: 'AZAffinity'` routes reads to same-AZ replicas ([Usage](docs/USAGE.md#cluster-mode))
+
+## Framework integrations
+
+| Package | Install | Setup |
+|---------|---------|-------|
+| [`@glidemq/hono`](https://github.com/avifenesh/glidemq-hono) | `npm i @glidemq/hono` | `app.use(glideMQ({ connection, queues: { ... } }))` |
+| [`@glidemq/fastify`](https://github.com/avifenesh/glidemq-fastify) | `npm i @glidemq/fastify` | `app.register(glideMQPlugin, { connection, queues: { ... } })` |
+| [`@glidemq/nestjs`](https://github.com/avifenesh/glidemq-nestjs) | `npm i @glidemq/nestjs` | `GlideMQModule.forRoot({ connection, queues: { ... } })` |
+| [`@glidemq/dashboard`](https://github.com/avifenesh/glidemq-dashboard) | `npm i @glidemq/dashboard` | `app.use('/dashboard', createDashboard([queue1, queue2]))` |
+| @glidemq/hapi | coming soon | Hapi plugin with the same REST + SSE surface |
+
+All framework packages provide REST endpoints, SSE events, and serverless Producer support. See each package's README for full documentation.
+
+## Cross-language
+
+Non-Node.js services can enqueue jobs into glide-mq queues using the HTTP proxy or direct FCALL:
+
+```typescript
+import { createProxyServer } from 'glide-mq/proxy';
+
+const proxy = createProxyServer({
+  connection: { addresses: [{ host: 'localhost', port: 6379 }] },
+  queues: ['emails', 'reports'],
+});
+proxy.app.listen(3000);
+```
+
+```bash
+curl -X POST http://localhost:3000/queues/emails/jobs \
+  -H 'Content-Type: application/json' \
+  -d '{"name": "send-email", "data": {"to": "user@example.com"}}'
+```
+
+Endpoints: `POST /queues/:name/jobs`, `POST /queues/:name/jobs/bulk`, `GET /queues/:name/jobs/:id`, `POST /queues/:name/pause`, `POST /queues/:name/resume`, `GET /queues/:name/counts`, `GET /health`.
+
+For zero-overhead integration, call Valkey Server Functions directly from any language with a Valkey client. See [Wire Protocol](docs/WIRE_PROTOCOL.md) for FCALL signatures, key layout, and examples in Python and Go.
+
 ## Documentation
 
-| Guide | What you'll learn |
-|-------|-------------------|
-| [Usage](docs/USAGE.md) | Queue & Worker basics, graceful shutdown, cluster mode |
-| [Advanced](docs/ADVANCED.md) | Schedulers, rate limiting, dedup, compression, retries & DLQ |
-| [Workflows](docs/WORKFLOWS.md) | FlowProducer, `chain`, `group`, `chord` pipelines |
-| [Observability](docs/OBSERVABILITY.md) | OpenTelemetry, job logs, `@glidemq/dashboard` |
-| [Testing](docs/TESTING.md) | In-memory `TestQueue` & `TestWorker` — no Valkey needed |
-| [Architecture](docs/ARCHITECTURE.md) | Key design, Valkey functions, data layout |
-| [Migration](docs/MIGRATION.md) | Coming from BullMQ? API mapping & workarounds |
-
-## Get Involved
-
-- ⭐ [Star on GitHub](https://github.com/avifenesh/glide-mq) — helps others find the project
-- 🐛 [Open an issue](https://github.com/avifenesh/glide-mq/issues) — bug reports & feature requests welcome
-- 💬 [Discussions](https://github.com/avifenesh/glide-mq/discussions) — questions, ideas, show & tell
+| Guide | Topics |
+|-------|--------|
+| [Usage](docs/USAGE.md) | Queue, Worker, Broadcast, Producer, batch, request-reply, step jobs, graceful shutdown, cluster mode |
+| [Advanced](docs/ADVANCED.md) | Schedulers, rate limiting, dedup, compression, retries, DLQ, custom IDs, LIFO, TTL, serializers |
+| [Workflows](docs/WORKFLOWS.md) | FlowProducer, DAG, `chain`, `group`, `chord`, dynamic children |
+| [Observability](docs/OBSERVABILITY.md) | OpenTelemetry, time-series metrics, job logs, dashboard |
+| [Serverless](docs/SERVERLESS.md) | Producer, ServerlessPool, Lambda and Edge deployment |
+| [Testing](docs/TESTING.md) | In-memory `TestQueue` and `TestWorker` -- no Valkey needed |
+| [Wire Protocol](docs/WIRE_PROTOCOL.md) | Cross-language FCALL specs, key layout, Python and Go examples |
+| [Architecture](docs/ARCHITECTURE.md) | Key design, Valkey functions, LIFO, Broadcast, DAG internals |
+| [Durability](docs/DURABILITY.md) | Persistence modes, crash windows, feature-specific durability |
+| [Migration](docs/MIGRATION.md) | Coming from BullMQ? API mapping and step-by-step guide |
+
+## Limitations
+
+- Requires a running Valkey 7.0+ or Redis 7.0+ instance. There is no embedded mode.
+- Node.js only. The Rust-native NAPI client (`@valkey/valkey-glide`) does not run in browsers or Deno.
+- At-least-once delivery semantics. Jobs may be processed more than once after crashes or stalled recovery.
+- Not a streaming platform. glide-mq is a job/task queue, not a replacement for Kafka or NATS JetStream.
+- Single dependency on `@glidemq/speedkey` (which wraps `@valkey/valkey-glide`). Native addon compilation is required on install.
+
+## Ecosystem
+
+| Package | Description | Links |
+|---------|-------------|-------|
+| [glide-mq](https://github.com/avifenesh/glide-mq) | Core queue library | [npm](https://www.npmjs.com/package/glide-mq) |
+| [@glidemq/hono](https://github.com/avifenesh/glidemq-hono) | Hono middleware -- REST endpoints, SSE, serverless Producer | [npm](https://www.npmjs.com/package/@glidemq/hono) |
+| [@glidemq/fastify](https://github.com/avifenesh/glidemq-fastify) | Fastify plugin -- REST endpoints, SSE, serverless Producer | [npm](https://www.npmjs.com/package/@glidemq/fastify) |
+| [@glidemq/nestjs](https://github.com/avifenesh/glidemq-nestjs) | NestJS module -- decorators, DI, lifecycle management | [npm](https://www.npmjs.com/package/@glidemq/nestjs) |
+| [@glidemq/dashboard](https://github.com/avifenesh/glidemq-dashboard) | Web UI -- metrics charts, scheduler management, job mutations | [npm](https://www.npmjs.com/package/@glidemq/dashboard) |
+| [@glidemq/speedkey](https://github.com/avifenesh/speedkey) | Valkey GLIDE client with native NAPI bindings | [npm](https://www.npmjs.com/package/@glidemq/speedkey) |
+| [glidemq-examples](https://github.com/avifenesh/glidemq-examples) | 34 runnable examples across frameworks and use cases | [GitHub](https://github.com/avifenesh/glidemq-examples) |
+
+> If glide-mq is useful to you, consider [starring the repo](https://github.com/avifenesh/glide-mq). It helps others find the project.
+
+## Contributing
+
+Bug reports, feature requests, and pull requests are welcome. See [CHANGELOG.md](CHANGELOG.md) for release history.
+
+- [Open an issue](https://github.com/avifenesh/glide-mq/issues)
+- [Discussions](https://github.com/avifenesh/glide-mq/discussions)
 
 ## License
 

package/dist/base-worker.d.ts

@@ -0,0 +1,282 @@
+import { EventEmitter } from 'events';
+import type { WorkerOptions, Processor, BatchProcessor, Client, Serializer } from './types';
+import { Job } from './job';
+import { buildKeys } from './utils';
+import type { QueueKeys } from './functions/index';
+import { Scheduler } from './scheduler';
+export type WorkerEvent = 'completed' | 'failed' | 'error' | 'stalled' | 'closing' | 'closed' | 'active' | 'drained';
+/**
+ * Configuration that differs between Worker and BroadcastWorker.
+ * Passed from the subclass constructor to BaseWorker.
+ */
+export interface BaseWorkerConfig {
+    /** Consumer group name for XREADGROUP / stalled reclaim. */
+    consumerGroup: string;
+    /** Whether this worker operates in broadcast (fan-out) mode. */
+    broadcastMode: boolean;
+    /** Stream ID to start from when creating the consumer group. */
+    startFrom: string;
+}
+/**
+ * Base class that contains all shared Worker / BroadcastWorker logic.
+ *
+ * Subclasses implement:
+ * - pollOnce(): the per-iteration poll strategy (list polling vs stream-only)
+ * - getAttemptsMade(): how to resolve attemptsMade (shared hash vs per-subscription)
+ * - isDrainComplete(): what "empty" means for drain()
+ */
+export declare abstract class BaseWorker<D = any, R = any> extends EventEmitter {
+    readonly name: string;
+    protected opts: WorkerOptions;
+    protected processor: Processor<D, R>;
+    protected commandClient: Client | null;
+    protected commandClientOwned: boolean;
+    protected blockingClient: Client | null;
+    protected running: boolean;
+    protected paused: boolean;
+    protected closing: boolean;
+    protected closed: boolean;
+    protected queueKeys: ReturnType<typeof buildKeys>;
+    protected consumerId: string;
+    protected activeCount: number;
+    protected activePromises: Set<Promise<void>>;
+    protected activeAbortControllers: Map<string, AbortController>;
+    protected scheduler: Scheduler | null;
+    protected initPromise: Promise<void>;
+    protected rateLimitUntil: number;
+    protected isDrained: boolean;
+    protected reconnectBackoff: number;
+    protected internalEvents: EventEmitter<[never]>;
+    protected concurrency: number;
+    protected prefetch: number;
+    protected blockTimeout: number;
+    protected stalledInterval: number;
+    protected maxStalledCount: number;
+    protected lockDuration: number;
+    protected heartbeatIntervals: Map<string, ReturnType<typeof setInterval>>;
+    protected xreadStreams: Record<string, string>;
+    protected globalConcurrencyEnabled: boolean;
+    protected globalRateLimitEnabled: boolean;
+    protected cachedRateLimitMax: number;
+    protected cachedRateLimitDuration: number;
+    protected sandboxClose?: (force?: boolean) => Promise<void>;
+    protected workerHeartbeatTimer: ReturnType<typeof setInterval> | null;
+    protected pollLoopPromise: Promise<void> | null;
+    protected readonly startedAt: number;
+    protected readonly hostname: string;
+    protected serializer: Serializer;
+    protected readonly batchMode: boolean;
+    protected readonly batchSize: number;
+    protected readonly batchTimeout: number;
+    protected readonly batchProcessor: BatchProcessor<D, R> | null;
+    protected readonly consumerGroup: string;
+    protected readonly broadcastMode: boolean;
+    protected readonly startFrom: string;
+    protected constructor(name: string, processor: Processor<D, R> | BatchProcessor<D, R> | string, opts: WorkerOptions, config: BaseWorkerConfig);
+    /**
+     * Wait for the worker to be fully initialized and connected.
+     */
+    waitUntilReady(): Promise<void>;
+    private init;
+    /**
+     * Main poll loop: XREADGROUP BLOCK on the stream, dispatch jobs to the processor.
+     * Respects concurrency limits by only requesting (prefetch - activeCount) entries.
+     * On connection errors, uses exponential backoff (1s, 2s, 4s, 8s, max 30s) and reconnects.
+     */
+    protected pollLoop(): Promise<void>;
+    private reconnectCtx;
+    /**
+     * Attempt to reconnect clients and resume polling after a connection error.
+     */
+    private reconnectAndResume;
+    protected waitForSlot(): Promise<void>;
+    /**
+     * Subclass-specific polling strategy. Called once per poll loop iteration.
+     * Worker: checks priority/LIFO lists then XREADGROUP.
+     * BroadcastWorker: XREADGROUP only.
+     */
+    protected abstract pollOnce(): Promise<void>;
+    /**
+     * Dispatch a single job for processing.
+     * Increments activeCount, runs the processor, then completes or fails the job.
+     */
+    protected dispatchJob(jobId: string, entryId: string): void;
+    /**
+     * Dispatch a batch for processing (c>1 mode).
+     */
+    protected dispatchBatch(batch: {
+        jobId: string;
+        entryId: string;
+        job: Job<D, R>;
+    }[]): void;
+    /**
+     * Activate collected batch entries and process them.
+     * Shared between Worker and BroadcastWorker batch paths.
+     */
+    protected activateAndProcessBatch(collected: {
+        jobId: string;
+        entryId: string;
+    }[]): Promise<void>;
+    /**
+     * Process a batch of jobs through the batch processor.
+     * Handles completion, failure, and partial failure (BatchError) for each job individually.
+     */
+    protected processBatch(batch: {
+        jobId: string;
+        entryId: string;
+        job: Job<D, R>;
+    }[]): Promise<void>;
+    /**
+     * Handle a moveToActive result that is not a valid hash (null or REVOKED).
+     * Returns true if the result was handled (caller should return), false if the hash is valid.
+     */
+    protected handleMoveToActiveEdgeCase(moveResult: Record<string, string> | 'REVOKED' | 'EXPIRED' | 'GROUP_FULL' | 'GROUP_RATE_LIMITED' | 'GROUP_TOKEN_LIMITED' | 'ERR:COST_EXCEEDS_CAPACITY' | null, jobId: string, entryId: string): Promise<boolean>;
+    /**
+     * Run the processor with optional timeout, AbortController, and heartbeat.
+     * Returns { result, error } - exactly one will be set.
+     */
+    protected runProcessor(job: Job<D, R>, jobId: string): Promise<{
+        result?: R;
+        error?: Error;
+        aborted: boolean;
+    }>;
+    /**
+     * Resolve the number of attempts made for a job.
+     * Worker: reads from job.attemptsMade (shared hash).
+     * BroadcastWorker: reads per-subscription counter from :sub: hash.
+     */
+    protected getAttemptsMade(job: Job<D, R>, _jobId: string): Promise<number>;
+    /**
+     * Handle a failed job: applies rate limiting, backoff, DLQ, and emits 'failed'.
+     * Returns true when the job reached a terminal failed state, false when it will retry.
+     */
+    protected handleJobFailure(job: Job<D, R>, jobId: string, entryId: string, error: Error): Promise<boolean>;
+    /**
+     * Move an active job back into delayed state after the processor requests a pause.
+     */
+    protected handleMoveToDelayed(job: Job<D, R>, jobId: string, entryId: string, request: {
+        delayedUntil: number;
+        serializedData?: string;
+        nextData?: D;
+    }): Promise<void>;
+    /**
+     * After a repeatAfterComplete job completes or terminally fails,
+     * update the scheduler entry so the next job is scheduled.
+     *
+     * KNOWN LIMITATIONS:
+     * 1. Non-atomic: This update happens after the job completion transaction,
+     *    so a worker crash between completion and this call will leave the scheduler
+     *    stuck at nextRun=0 (awaiting completion sentinel) indefinitely.
+     * 2. Non-worker failures: Jobs that reach terminal failure outside the worker
+     *    path (e.g., revoked jobs, expired jobs in moveToActive, stalled terminal
+     *    failures in glidemq_reclaimStalled) never trigger this update, leaving
+     *    the scheduler permanently stuck.
+     * 3. Race conditions: The idempotency check (nextRun === 0) prevents duplicate
+     *    updates from stalled reclaim, but doesn't prevent races with concurrent
+     *    upsertJobScheduler/removeJobScheduler (those use scheduler lock, this doesn't).
+     *
+     * MITIGATION: Run multiple workers for redundancy. Manually remove/re-add the
+     * scheduler to recover from stuck state.
+     *
+     * FUTURE WORK: Move scheduler update into Lua completion/failure functions to
+     * make it atomic and handle all terminal failure paths.
+     */
+    protected updateSchedulerAfterComplete(schedulerName: string, now: number): Promise<void>;
+    /**
+     * Build parent dependency info for complete/completeAndFetchNext calls.
+     */
+    protected buildParentInfo(job: Job<D, R>, jobId: string): Promise<{
+        depsMember: string;
+        parentId: string;
+        parentKeys: QueueKeys;
+    } | undefined>;
+    protected orderingMetaField(job: Job<D, R>): string | null;
+    /**
+     * Checks whether this job can run now under per-key ordering.
+     * Returns false when an earlier sequence for the same key is still pending.
+     */
+    protected isOrderingTurn(job: Job<D, R>): Promise<boolean>;
+    /**
+     * Re-enqueue out-of-order jobs instead of holding an active slot.
+     */
+    protected deferOutOfOrderJob(jobId: string, entryId: string): Promise<void>;
+    /**
+     * Process a job through its full lifecycle: activate, run processor, complete, fetch next.
+     * Used for both c=1 (inline, blocking poll loop) and c>1 (dispatched via dispatchJob).
+     * Chains into the next job via completeAndFetchNext to reuse the same dispatch slot.
+     */
+    protected processJob(jobId: string, entryId: string): Promise<void>;
+    /**
+     * Abort a job that is currently being processed by this worker.
+     * The processor receives the abort signal via job.abortSignal and must check it cooperatively.
+     * Returns true if the job was found and aborted, false if not currently active.
+     */
+    abortJob(jobId: string): boolean;
+    protected startHeartbeat(jobId: string): void;
+    protected stopHeartbeat(jobId: string): void;
+    protected moveToDLQ(job: Job<D, R>, error: Error): Promise<void>;
+    /**
+     * Check the server-side rate limiter and wait if the limit is exceeded.
+     * Also respects any manual rate limit set via rateLimit(ms).
+     */
+    protected waitForRateLimit(): Promise<void>;
+    /** Refresh cached meta flags from Valkey. Called on init and each scheduler tick. */
+    private refreshMetaFlags;
+    /**
+     * Register this worker in Valkey with a TTL-based heartbeat key.
+     * The key expires after stalledInterval ms; a periodic timer refreshes it at half that interval.
+     * Registration failure is non-fatal - the worker can still process jobs.
+     */
+    private registerWorker;
+    /**
+     * Check if the worker is currently running and not paused.
+     */
+    isRunning(): boolean;
+    /**
+     * Check if the worker is currently paused.
+     */
+    isPaused(): boolean;
+    /**
+     * Manually trigger a rate limit pause for the given duration.
+     * Subsequent jobs will wait until the pause expires.
+     */
+    rateLimit(ms: number): Promise<void>;
+    /**
+     * Pause the worker. If force=false (default), waits for active jobs to finish.
+     */
+    pause(force?: boolean): Promise<void>;
+    /**
+     * Resume the worker after a pause.
+     */
+    resume(): Promise<void>;
+    /**
+     * Check whether the queue has been fully drained.
+     * Worker: stream + scheduled must both be empty.
+     * BroadcastWorker: scheduled only (stream entries are retained for fan-out).
+     */
+    protected isDrainComplete(): Promise<boolean>;
+    /**
+     * Process all remaining jobs in the queue, then stop gracefully.
+     * Keeps polling until isDrainComplete() returns true, then closes the worker.
+     */
+    drain(): Promise<void>;
+    /**
+     * Close the worker. If force=false (default), waits for active jobs to finish.
+     * Idempotent: safe to call multiple times.
+     */
+    close(force?: boolean): Promise<void>;
+    protected waitForActiveJobs(): Promise<void>;
+    static isRateLimitError(error: Error): boolean;
+    static RateLimitError: {
+        new (): {
+            name: string;
+            message: string;
+            stack?: string;
+            cause?: unknown;
+        };
+        captureStackTrace(targetObject: object, constructorOpt?: Function): void;
+        prepareStackTrace(err: Error, stackTraces: NodeJS.CallSite[]): any;
+        stackTraceLimit: number;
+    };
+}
+//# sourceMappingURL=base-worker.d.ts.map