glide-mq 0.1.0

package/README.md

@@ -0,0 +1,190 @@
+# glide-mq
+
+High-performance message queue for Node.js, built on Valkey/Redis Streams with [Valkey GLIDE](https://github.com/valkey-io/valkey-glide) native NAPI bindings.
+
+## Performance
+
+| Concurrency | Throughput |
+|-------------|-----------|
+| c=1 | 4,376 jobs/s |
+| c=10 | 20,979 jobs/s |
+| c=50 | 44,643 jobs/s |
+
+No-op processor, Valkey 8.0, single node.
+
+## Why
+
+- **Streams-first** - uses Redis Streams + consumer groups + PEL instead of Lists + BRPOPLPUSH. Fewer moving parts, built-in at-least-once delivery.
+- **Server Functions** - single `FUNCTION LOAD` instead of dozens of EVAL scripts. Persistent across restarts, no NOSCRIPT cache-miss errors.
+- **1 RTT per job** - `completeAndFetchNext` combines job completion + next job fetch + activation in a single FCALL round trip.
+- **Cluster-native** - hash-tagged keys from day one. No afterthought `{braces}` requirement.
+- **Native bindings** - built on [@glidemq/speedkey](https://github.com/avifenesh/speedkey) (valkey-glide with Rust core + NAPI).
+
+## Install
+
+```bash
+npm install glide-mq
+```
+
+Requires Node.js 20+ and a running Valkey (7.2+) or Redis (6.2+) instance.
+
+## Quick Start
+
+```typescript
+import { Queue, Worker } from 'glide-mq';
+
+const connection = { addresses: [{ host: 'localhost', port: 6379 }] };
+
+// Producer
+const queue = new Queue('tasks', { connection });
+await queue.add('send-email', { to: 'user@example.com', subject: 'Hello' });
+
+// Consumer
+const worker = new Worker('tasks', async (job) => {
+  console.log(`Processing ${job.name}: ${JSON.stringify(job.data)}`);
+  return { sent: true };
+}, { connection, concurrency: 10 });
+
+worker.on('completed', (job) => console.log(`Job ${job.id} completed`));
+worker.on('failed', (job, err) => console.log(`Job ${job.id} failed: ${err.message}`));
+```
+
+## Features
+
+### Core
+- **Queue** - add, addBulk, getJob, getJobs, pause, resume, obliterate, drain
+- **Worker** - concurrent processing, XREADGROUP polling, graceful shutdown
+- **Job** - progress, updateData, retry, remove, log, state queries, waitUntilFinished
+
+### Reliability
+- **Retries** - fixed, exponential, jitter backoff + custom strategy functions
+- **Stalled recovery** - XAUTOCLAIM with heartbeat-based lock renewal
+- **Per-job timeout** - automatic failure if processor exceeds timeout
+- **Dead letter queue** - permanently failed jobs routed to a separate queue
+
+### Advanced
+- **Deduplication** - simple, throttle, debounce modes
+- **Rate limiting** - sliding window, per-queue
+- **Global concurrency** - limit active jobs across all workers
+- **Job retention** - removeOnComplete/removeOnFail (count, age-based)
+- **Priorities** - encoded in sorted set scores, FIFO within same priority
+
+### Workflows
+- **FlowProducer** - atomic parent-child job trees with nested flows
+- **chain(queue, jobs)** - sequential pipeline, each job receives previous result
+- **group(queue, jobs)** - parallel execution, parent completes when all children done
+- **chord(queue, group, callback)** - run group, then callback with all results
+
+### Observability
+- **QueueEvents** - stream-based event subscription (added, completed, failed, stalled, etc.)
+- **Job schedulers** - cron patterns and fixed intervals for repeatable jobs
+- **Metrics** - getJobCounts, getMetrics
+- **OpenTelemetry** - optional spans for Queue.add, Worker.process, FlowProducer.add
+
+### Operations
+- **Graceful shutdown** - SIGTERM/SIGINT handler, waits for active jobs
+- **Connection recovery** - exponential backoff reconnect with function library reload
+- **Job revocation** - cooperative cancellation via AbortSignal
+- **Cluster mode** - all features work in Valkey Cluster (pass `clusterMode: true`)
+
+## API
+
+### Queue
+
+```typescript
+const queue = new Queue('name', {
+  connection: { addresses: [{ host, port }], clusterMode: false },
+  prefix: 'glide', // key prefix (default: 'glide')
+});
+
+await queue.add('jobName', data, {
+  delay: 5000,        // delayed job (ms)
+  priority: 1,        // lower = higher priority
+  attempts: 3,        // max retry attempts
+  backoff: { type: 'exponential', delay: 1000 },
+  timeout: 30000,     // per-job timeout (ms)
+  removeOnComplete: true,  // or count, or { age, count }
+  deduplication: { id: 'unique-key', mode: 'simple' },
+});
+
+await queue.pause();
+await queue.resume();
+await queue.getJobCounts();  // { waiting, active, delayed, completed, failed }
+await queue.obliterate({ force: true });
+await queue.close();
+```
+
+### Worker
+
+```typescript
+const worker = new Worker('name', async (job) => {
+  await job.updateProgress(50);
+  return result;
+}, {
+  connection,
+  concurrency: 10,
+  blockTimeout: 5000,
+  stalledInterval: 30000,
+  lockDuration: 30000,
+  limiter: { max: 100, duration: 60000 },
+});
+
+worker.on('completed', (job, result) => {});
+worker.on('failed', (job, error) => {});
+await worker.close();
+```
+
+### Flows
+
+```typescript
+import { FlowProducer, chain, group, chord } from 'glide-mq';
+
+// Parent-child
+const flow = new FlowProducer({ connection });
+await flow.add({
+  name: 'parent', queueName: 'tasks', data: {},
+  children: [
+    { name: 'child1', queueName: 'tasks', data: {} },
+    { name: 'child2', queueName: 'tasks', data: {} },
+  ],
+});
+
+// Chain: A -> B -> C (sequential)
+await chain(connection, 'tasks', [
+  { name: 'step1', data: {} },
+  { name: 'step2', data: {} },
+]);
+
+// Group: A + B + C (parallel)
+await group(connection, 'tasks', [
+  { name: 'task1', data: {} },
+  { name: 'task2', data: {} },
+]);
+```
+
+### Events
+
+```typescript
+import { QueueEvents } from 'glide-mq';
+
+const events = new QueueEvents('tasks', { connection });
+events.on('completed', ({ jobId, returnvalue }) => {});
+events.on('failed', ({ jobId, failedReason }) => {});
+events.on('stalled', ({ jobId }) => {});
+```
+
+## Cluster Mode
+
+```typescript
+const connection = {
+  addresses: [{ host: 'cluster-node', port: 7000 }],
+  clusterMode: true,
+};
+
+// Everything works the same - keys are hash-tagged automatically
+const queue = new Queue('tasks', { connection });
+```
+
+## License
+
+Apache-2.0

package/dist/connection.d.ts

@@ -0,0 +1,36 @@
+import type { ConnectionOptions, Client } from './types';
+/**
+ * Create a GlideClient (standalone) or GlideClusterClient (cluster) based on options.
+ */
+export declare function createClient(opts: ConnectionOptions): Promise<Client>;
+/**
+ * Create a dedicated client for XREADGROUP BLOCK / XREAD BLOCK.
+ * Must not be shared with non-blocking commands.
+ */
+export declare function createBlockingClient(opts: ConnectionOptions): Promise<Client>;
+/**
+ * Ensure the glidemq function library is loaded and up-to-date on the server.
+ *
+ * Strategy:
+ * 1. Try FCALL glidemq_version with empty keys/args
+ * 2. If succeeds and version matches, return (already loaded)
+ * 3. If fails (function not found) or version mismatch, call functionLoad with replace
+ * 4. For cluster clients, load to all primaries via route option
+ *
+ * @param client - The Valkey client
+ * @param librarySource - The Lua library source code to load
+ * @param clusterMode - Whether the client is a cluster client (determines routing for FUNCTION LOAD)
+ */
+export declare function ensureFunctionLibrary(client: Client, librarySource?: string, clusterMode?: boolean): Promise<void>;
+/**
+ * Create a consumer group for the given stream key.
+ * Uses XGROUP CREATE with mkStream to auto-create the stream if it doesn't exist.
+ * Handles BUSYGROUP error (group already exists) gracefully.
+ *
+ * @param client - The Valkey client
+ * @param streamKey - The stream key to create the group on
+ * @param groupName - The consumer group name
+ * @param startId - The ID to start reading from (default: '0' = beginning)
+ */
+export declare function createConsumerGroup(client: Client, streamKey: string, groupName: string, startId?: string): Promise<void>;
+//# sourceMappingURL=connection.d.ts.map

package/dist/connection.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"connection.d.ts","sourceRoot":"","sources":["../src/connection.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,iBAAiB,EAAE,MAAM,EAAE,MAAM,SAAS,CAAC;AAIzD;;GAEG;AACH,wBAAsB,YAAY,CAAC,IAAI,EAAE,iBAAiB,GAAG,OAAO,CAAC,MAAM,CAAC,CAiB3E;AAED;;;GAGG;AACH,wBAAsB,oBAAoB,CAAC,IAAI,EAAE,iBAAiB,GAAG,OAAO,CAAC,MAAM,CAAC,CAEnF;AAED;;;;;;;;;;;;GAYG;AACH,wBAAsB,qBAAqB,CACzC,MAAM,EAAE,MAAM,EACd,aAAa,GAAE,MAAuB,EACtC,WAAW,GAAE,OAAe,GAC3B,OAAO,CAAC,IAAI,CAAC,CA0Bf;AAED;;;;;;;;;GASG;AACH,wBAAsB,mBAAmB,CACvC,MAAM,EAAE,MAAM,EACd,SAAS,EAAE,MAAM,EACjB,SAAS,EAAE,MAAM,EACjB,OAAO,GAAE,MAAY,GACpB,OAAO,CAAC,IAAI,CAAC,CAYf"}

package/dist/connection.js

@@ -0,0 +1,100 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.createClient = createClient;
+exports.createBlockingClient = createBlockingClient;
+exports.ensureFunctionLibrary = ensureFunctionLibrary;
+exports.createConsumerGroup = createConsumerGroup;
+const speedkey_1 = require("@glidemq/speedkey");
+const errors_1 = require("./errors");
+const index_1 = require("./functions/index");
+/**
+ * Create a GlideClient (standalone) or GlideClusterClient (cluster) based on options.
+ */
+async function createClient(opts) {
+    const config = {
+        addresses: opts.addresses,
+        useTLS: opts.useTLS,
+        credentials: opts.credentials,
+    };
+    try {
+        if (opts.clusterMode) {
+            return await speedkey_1.GlideClusterClient.createClient(config);
+        }
+        return await speedkey_1.GlideClient.createClient(config);
+    }
+    catch (err) {
+        throw new errors_1.ConnectionError(`Failed to create ${opts.clusterMode ? 'cluster' : 'standalone'} client: ${err instanceof Error ? err.message : String(err)}`);
+    }
+}
+/**
+ * Create a dedicated client for XREADGROUP BLOCK / XREAD BLOCK.
+ * Must not be shared with non-blocking commands.
+ */
+async function createBlockingClient(opts) {
+    return createClient(opts);
+}
+/**
+ * Ensure the glidemq function library is loaded and up-to-date on the server.
+ *
+ * Strategy:
+ * 1. Try FCALL glidemq_version with empty keys/args
+ * 2. If succeeds and version matches, return (already loaded)
+ * 3. If fails (function not found) or version mismatch, call functionLoad with replace
+ * 4. For cluster clients, load to all primaries via route option
+ *
+ * @param client - The Valkey client
+ * @param librarySource - The Lua library source code to load
+ * @param clusterMode - Whether the client is a cluster client (determines routing for FUNCTION LOAD)
+ */
+async function ensureFunctionLibrary(client, librarySource = index_1.LIBRARY_SOURCE, clusterMode = false) {
+    if (clusterMode) {
+        // In cluster mode, always load with REPLACE to all primaries.
+        // FUNCTION LOAD REPLACE is idempotent - safe to call every time.
+        // Skipping the version check avoids FCALL routing to replicas (empty keys = random node).
+        await client.functionLoad(librarySource, {
+            replace: true,
+            route: 'allPrimaries',
+        });
+        return;
+    }
+    // Standalone: check version first to avoid unnecessary reload
+    try {
+        const result = await client.fcall('glidemq_version', [], []);
+        if (String(result) === index_1.LIBRARY_VERSION)
+            return;
+    }
+    catch (err) {
+        const msg = err instanceof Error ? err.message : String(err);
+        if (!msg.includes('Function not loaded') && !msg.includes('Function not found') && !msg.includes('No matching script') && !msg.includes('NOSCRIPT')) {
+            throw err;
+        }
+    }
+    await client.functionLoad(librarySource, {
+        replace: true,
+    });
+}
+/**
+ * Create a consumer group for the given stream key.
+ * Uses XGROUP CREATE with mkStream to auto-create the stream if it doesn't exist.
+ * Handles BUSYGROUP error (group already exists) gracefully.
+ *
+ * @param client - The Valkey client
+ * @param streamKey - The stream key to create the group on
+ * @param groupName - The consumer group name
+ * @param startId - The ID to start reading from (default: '0' = beginning)
+ */
+async function createConsumerGroup(client, streamKey, groupName, startId = '0') {
+    try {
+        await client.xgroupCreate(streamKey, groupName, startId, {
+            mkStream: true,
+        });
+    }
+    catch (err) {
+        // BUSYGROUP means the group already exists - that's fine
+        if (err instanceof Error && err.message.includes('BUSYGROUP')) {
+            return;
+        }
+        throw err;
+    }
+}
+//# sourceMappingURL=connection.js.map

package/dist/connection.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"connection.js","sourceRoot":"","sources":["../src/connection.ts"],"names":[],"mappings":";;AAQA,oCAiBC;AAMD,oDAEC;AAeD,sDA8BC;AAYD,kDAiBC;AA3GD,gDAAoE;AAEpE,qCAA2C;AAC3C,6CAAoE;AAEpE;;GAEG;AACI,KAAK,UAAU,YAAY,CAAC,IAAuB;IACxD,MAAM,MAAM,GAAG;QACb,SAAS,EAAE,IAAI,CAAC,SAAS;QACzB,MAAM,EAAE,IAAI,CAAC,MAAM;QACnB,WAAW,EAAE,IAAI,CAAC,WAAW;KAC9B,CAAC;IAEF,IAAI,CAAC;QACH,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;YACrB,OAAO,MAAM,6BAAkB,CAAC,YAAY,CAAC,MAAM,CAAC,CAAC;QACvD,CAAC;QACD,OAAO,MAAM,sBAAW,CAAC,YAAY,CAAC,MAAM,CAAC,CAAC;IAChD,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACb,MAAM,IAAI,wBAAe,CACvB,oBAAoB,IAAI,CAAC,WAAW,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,YAAY,YAAY,GAAG,YAAY,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,EAAE,CAC9H,CAAC;IACJ,CAAC;AACH,CAAC;AAED;;;GAGG;AACI,KAAK,UAAU,oBAAoB,CAAC,IAAuB;IAChE,OAAO,YAAY,CAAC,IAAI,CAAC,CAAC;AAC5B,CAAC;AAED;;;;;;;;;;;;GAYG;AACI,KAAK,UAAU,qBAAqB,CACzC,MAAc,EACd,gBAAwB,sBAAc,EACtC,cAAuB,KAAK;IAE5B,IAAI,WAAW,EAAE,CAAC;QAChB,8DAA8D;QAC9D,iEAAiE;QACjE,0FAA0F;QAC1F,MAAO,MAA6B,CAAC,YAAY,CAAC,aAAa,EAAE;YAC/D,OAAO,EAAE,IAAI;YACb,KAAK,EAAE,cAAc;SACtB,CAAC,CAAC;QACH,OAAO;IACT,CAAC;IAED,8DAA8D;IAC9D,IAAI,CAAC;QACH,MAAM,MAAM,GAAG,MAAM,MAAM,CAAC,KAAK,CAAC,iBAAiB,EAAE,EAAE,EAAE,EAAE,CAAC,CAAC;QAC7D,IAAI,MAAM,CAAC,MAAM,CAAC,KAAK,uBAAe;YAAE,OAAO;IACjD,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACb,MAAM,GAAG,GAAG,GAAG,YAAY,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;QAC7D,IAAI,CAAC,GAAG,CAAC,QAAQ,CAAC,qBAAqB,CAAC,IAAI,CAAC,GAAG,CAAC,QAAQ,CAAC,oBAAoB,CAAC,IAAI,CAAC,GAAG,CAAC,QAAQ,CAAC,oBAAoB,CAAC,IAAI,CAAC,GAAG,CAAC,QAAQ,CAAC,UAAU,CAAC,EAAE,CAAC;YACpJ,MAAM,GAAG,CAAC;QACZ,CAAC;IACH,CAAC;IAED,MAAO,MAAsB,CAAC,YAAY,CAAC,aAAa,EAAE;QACxD,OAAO,EAAE,IAAI;KACd,CAAC,CAAC;AACL,CAAC;AAED;;;;;;;;;GASG;AACI,KAAK,UAAU,mBAAmB,CACvC,MAAc,EACd,SAAiB,EACjB,SAAiB,EACjB,UAAkB,GAAG;IAErB,IAAI,CAAC;QACH,MAAM,MAAM,CAAC,YAAY,CAAC,SAAS,EAAE,SAAS,EAAE,OAAO,EAAE;YACvD,QAAQ,EAAE,IAAI;SACf,CAAC,CAAC;IACL,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACb,yDAAyD;QACzD,IAAI,GAAG,YAAY,KAAK,IAAI,GAAG,CAAC,OAAO,CAAC,QAAQ,CAAC,WAAW,CAAC,EAAE,CAAC;YAC9D,OAAO;QACT,CAAC;QACD,MAAM,GAAG,CAAC;IACZ,CAAC;AACH,CAAC"}

package/dist/errors.d.ts

@@ -0,0 +1,10 @@
+export declare class GlideMQError extends Error {
+    constructor(message: string);
+}
+export declare class ConnectionError extends GlideMQError {
+    constructor(message: string);
+}
+export declare class ScriptError extends GlideMQError {
+    constructor(message: string);
+}
+//# sourceMappingURL=errors.d.ts.map

package/dist/errors.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"errors.d.ts","sourceRoot":"","sources":["../src/errors.ts"],"names":[],"mappings":"AAAA,qBAAa,YAAa,SAAQ,KAAK;gBACzB,OAAO,EAAE,MAAM;CAI5B;AAED,qBAAa,eAAgB,SAAQ,YAAY;gBACnC,OAAO,EAAE,MAAM;CAI5B;AAED,qBAAa,WAAY,SAAQ,YAAY;gBAC/B,OAAO,EAAE,MAAM;CAI5B"}

package/dist/errors.js

@@ -0,0 +1,25 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.ScriptError = exports.ConnectionError = exports.GlideMQError = void 0;
+class GlideMQError extends Error {
+    constructor(message) {
+        super(message);
+        this.name = 'GlideMQError';
+    }
+}
+exports.GlideMQError = GlideMQError;
+class ConnectionError extends GlideMQError {
+    constructor(message) {
+        super(message);
+        this.name = 'ConnectionError';
+    }
+}
+exports.ConnectionError = ConnectionError;
+class ScriptError extends GlideMQError {
+    constructor(message) {
+        super(message);
+        this.name = 'ScriptError';
+    }
+}
+exports.ScriptError = ScriptError;
+//# sourceMappingURL=errors.js.map

package/dist/errors.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"errors.js","sourceRoot":"","sources":["../src/errors.ts"],"names":[],"mappings":";;;AAAA,MAAa,YAAa,SAAQ,KAAK;IACrC,YAAY,OAAe;QACzB,KAAK,CAAC,OAAO,CAAC,CAAC;QACf,IAAI,CAAC,IAAI,GAAG,cAAc,CAAC;IAC7B,CAAC;CACF;AALD,oCAKC;AAED,MAAa,eAAgB,SAAQ,YAAY;IAC/C,YAAY,OAAe;QACzB,KAAK,CAAC,OAAO,CAAC,CAAC;QACf,IAAI,CAAC,IAAI,GAAG,iBAAiB,CAAC;IAChC,CAAC;CACF;AALD,0CAKC;AAED,MAAa,WAAY,SAAQ,YAAY;IAC3C,YAAY,OAAe;QACzB,KAAK,CAAC,OAAO,CAAC,CAAC;QACf,IAAI,CAAC,IAAI,GAAG,aAAa,CAAC;IAC5B,CAAC;CACF;AALD,kCAKC"}

package/dist/flow-producer.d.ts

@@ -0,0 +1,36 @@
+import type { FlowProducerOptions, FlowJob } from './types';
+import { Job } from './job';
+export interface JobNode {
+    job: Job;
+    children?: JobNode[];
+}
+export declare class FlowProducer {
+    private opts;
+    private client;
+    private closing;
+    constructor(opts: FlowProducerOptions);
+    /** @internal */
+    private getClient;
+    /**
+     * Add a flow (parent with children) atomically.
+     * Children can have their own children (recursive flows), which are flattened
+     * into multiple addFlow calls (one per level with children).
+     */
+    add(flow: FlowJob): Promise<JobNode>;
+    /**
+     * Add multiple independent flows.
+     */
+    addBulk(flows: FlowJob[]): Promise<JobNode[]>;
+    /**
+     * Recursively add a flow. If children themselves have children,
+     * those sub-flows are added first (bottom-up), and the resulting
+     * child jobs are used as direct children of the current parent.
+     */
+    private addFlowRecursive;
+    /**
+     * Close the FlowProducer and release the underlying client connection.
+     * Idempotent: safe to call multiple times.
+     */
+    close(): Promise<void>;
+}
+//# sourceMappingURL=flow-producer.d.ts.map

package/dist/flow-producer.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"flow-producer.d.ts","sourceRoot":"","sources":["../src/flow-producer.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,mBAAmB,EAAE,OAAO,EAAU,MAAM,SAAS,CAAC;AACpE,OAAO,EAAE,GAAG,EAAE,MAAM,OAAO,CAAC;AAM5B,MAAM,WAAW,OAAO;IACtB,GAAG,EAAE,GAAG,CAAC;IACT,QAAQ,CAAC,EAAE,OAAO,EAAE,CAAC;CACtB;AAED,qBAAa,YAAY;IACvB,OAAO,CAAC,IAAI,CAAsB;IAClC,OAAO,CAAC,MAAM,CAAuB;IACrC,OAAO,CAAC,OAAO,CAAS;gBAEZ,IAAI,EAAE,mBAAmB;IAIrC,gBAAgB;YACF,SAAS;IAYvB;;;;OAIG;IACG,GAAG,CAAC,IAAI,EAAE,OAAO,GAAG,OAAO,CAAC,OAAO,CAAC;IAe1C;;OAEG;IACG,OAAO,CAAC,KAAK,EAAE,OAAO,EAAE,GAAG,OAAO,CAAC,OAAO,EAAE,CAAC;IASnD;;;;OAIG;YACW,gBAAgB;IAyI9B;;;OAGG;IACG,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC;CAQ7B"}

package/dist/flow-producer.js

@@ -0,0 +1,185 @@
+"use strict";
+var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    var desc = Object.getOwnPropertyDescriptor(m, k);
+    if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) {
+      desc = { enumerable: true, get: function() { return m[k]; } };
+    }
+    Object.defineProperty(o, k2, desc);
+}) : (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    o[k2] = m[k];
+}));
+var __setModuleDefault = (this && this.__setModuleDefault) || (Object.create ? (function(o, v) {
+    Object.defineProperty(o, "default", { enumerable: true, value: v });
+}) : function(o, v) {
+    o["default"] = v;
+});
+var __importStar = (this && this.__importStar) || (function () {
+    var ownKeys = function(o) {
+        ownKeys = Object.getOwnPropertyNames || function (o) {
+            var ar = [];
+            for (var k in o) if (Object.prototype.hasOwnProperty.call(o, k)) ar[ar.length] = k;
+            return ar;
+        };
+        return ownKeys(o);
+    };
+    return function (mod) {
+        if (mod && mod.__esModule) return mod;
+        var result = {};
+        if (mod != null) for (var k = ownKeys(mod), i = 0; i < k.length; i++) if (k[i] !== "default") __createBinding(result, mod, k[i]);
+        __setModuleDefault(result, mod);
+        return result;
+    };
+})();
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.FlowProducer = void 0;
+const job_1 = require("./job");
+const utils_1 = require("./utils");
+const connection_1 = require("./connection");
+const index_1 = require("./functions/index");
+const telemetry_1 = require("./telemetry");
+class FlowProducer {
+    opts;
+    client = null;
+    closing = false;
+    constructor(opts) {
+        this.opts = opts;
+    }
+    /** @internal */
+    async getClient() {
+        if (!this.client) {
+            this.client = await (0, connection_1.createClient)(this.opts.connection);
+            await (0, connection_1.ensureFunctionLibrary)(this.client, index_1.LIBRARY_SOURCE, this.opts.connection.clusterMode ?? false);
+        }
+        return this.client;
+    }
+    /**
+     * Add a flow (parent with children) atomically.
+     * Children can have their own children (recursive flows), which are flattened
+     * into multiple addFlow calls (one per level with children).
+     */
+    async add(flow) {
+        return (0, telemetry_1.withSpan)('glide-mq.flow.add', {
+            'glide-mq.queue': flow.queueName,
+            'glide-mq.flow.name': flow.name,
+            'glide-mq.flow.childCount': flow.children?.length ?? 0,
+        }, async () => {
+            const client = await this.getClient();
+            return this.addFlowRecursive(client, flow);
+        });
+    }
+    /**
+     * Add multiple independent flows.
+     */
+    async addBulk(flows) {
+        const client = await this.getClient();
+        const results = [];
+        for (const flow of flows) {
+            results.push(await this.addFlowRecursive(client, flow));
+        }
+        return results;
+    }
+    /**
+     * Recursively add a flow. If children themselves have children,
+     * those sub-flows are added first (bottom-up), and the resulting
+     * child jobs are used as direct children of the current parent.
+     */
+    async addFlowRecursive(client, flow) {
+        const prefix = this.opts.prefix ?? 'glide';
+        const parentQueueName = flow.queueName;
+        const parentKeys = (0, utils_1.buildKeys)(parentQueueName, prefix);
+        // If no children, this is a leaf - add as a regular job (not a flow)
+        if (!flow.children || flow.children.length === 0) {
+            const { addJob } = await Promise.resolve().then(() => __importStar(require('./functions/index')));
+            const timestamp = Date.now();
+            const opts = flow.opts ?? {};
+            const jobId = await addJob(client, parentKeys, flow.name, JSON.stringify(flow.data), JSON.stringify(opts), timestamp, opts.delay ?? 0, opts.priority ?? 0, '', opts.attempts ?? 0);
+            const job = new job_1.Job(client, parentKeys, String(jobId), flow.name, flow.data, opts);
+            job.timestamp = timestamp;
+            return { job };
+        }
+        // Recursively process children that themselves have children (bottom-up).
+        const childNodeMap = new Map();
+        for (let i = 0; i < flow.children.length; i++) {
+            const child = flow.children[i];
+            if (child.children && child.children.length > 0) {
+                childNodeMap.set(i, await this.addFlowRecursive(client, child));
+            }
+        }
+        // Build leaf children data for addFlow (children without sub-children)
+        const childrenForLua = flow.children
+            .filter((_, i) => !childNodeMap.has(i))
+            .map((child) => {
+            const childOpts = child.opts ?? {};
+            return {
+                name: child.name,
+                data: JSON.stringify(child.data),
+                opts: JSON.stringify(childOpts),
+                delay: childOpts.delay ?? 0,
+                priority: childOpts.priority ?? 0,
+                maxAttempts: childOpts.attempts ?? 0,
+                keys: (0, utils_1.buildKeys)(child.queueName, prefix),
+                queuePrefix: (0, utils_1.keyPrefix)(prefix, child.queueName),
+                parentQueueName: parentQueueName,
+            };
+        });
+        // Build extra deps for sub-flow children (already created recursively)
+        const extraDeps = [];
+        for (const [i, subNode] of childNodeMap.entries()) {
+            const child = flow.children[i];
+            extraDeps.push(`${(0, utils_1.keyPrefix)(prefix, child.queueName)}:${subNode.job.id}`);
+        }
+        const timestamp = Date.now();
+        const parentOpts = flow.opts ?? {};
+        const ids = await (0, index_1.addFlow)(client, parentKeys, flow.name, JSON.stringify(flow.data), JSON.stringify(parentOpts), timestamp, parentOpts.delay ?? 0, parentOpts.priority ?? 0, parentOpts.attempts ?? 0, childrenForLua, extraDeps);
+        const parentId = ids[0];
+        // Set parentId and parentQueue on pre-existing sub-flow children
+        for (const [i, subNode] of childNodeMap.entries()) {
+            const child = flow.children[i];
+            const childKeys = (0, utils_1.buildKeys)(child.queueName, prefix);
+            await client.hset(childKeys.job(subNode.job.id), {
+                parentId: parentId,
+                parentQueue: parentQueueName,
+            });
+            subNode.job.parentId = parentId;
+            subNode.job.parentQueue = parentQueueName;
+        }
+        // Build JobNode tree - interleave sub-flow and leaf child nodes in original order
+        const childNodes = [];
+        let leafIdx = 0;
+        for (let i = 0; i < flow.children.length; i++) {
+            if (childNodeMap.has(i)) {
+                childNodes.push(childNodeMap.get(i));
+            }
+            else {
+                const childId = ids[1 + leafIdx];
+                const child = flow.children[i];
+                const childJob = new job_1.Job(client, (0, utils_1.buildKeys)(child.queueName, prefix), childId, child.name, child.data, child.opts ?? {});
+                childJob.timestamp = timestamp;
+                childJob.parentId = parentId;
+                childJob.parentQueue = parentQueueName;
+                childNodes.push({ job: childJob });
+                leafIdx++;
+            }
+        }
+        const parentJob = new job_1.Job(client, parentKeys, parentId, flow.name, flow.data, parentOpts);
+        parentJob.timestamp = timestamp;
+        return { job: parentJob, children: childNodes };
+    }
+    /**
+     * Close the FlowProducer and release the underlying client connection.
+     * Idempotent: safe to call multiple times.
+     */
+    async close() {
+        if (this.closing)
+            return;
+        this.closing = true;
+        if (this.client) {
+            this.client.close();
+            this.client = null;
+        }
+    }
+}
+exports.FlowProducer = FlowProducer;
+//# sourceMappingURL=flow-producer.js.map

package/dist/flow-producer.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"flow-producer.js","sourceRoot":"","sources":["../src/flow-producer.ts"],"names":[],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AACA,+BAA4B;AAC5B,mCAA+C;AAC/C,6CAAmE;AACnE,6CAA4D;AAC5D,2CAAuC;AAOvC,MAAa,YAAY;IACf,IAAI,CAAsB;IAC1B,MAAM,GAAkB,IAAI,CAAC;IAC7B,OAAO,GAAG,KAAK,CAAC;IAExB,YAAY,IAAyB;QACnC,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC;IACnB,CAAC;IAED,gBAAgB;IACR,KAAK,CAAC,SAAS;QACrB,IAAI,CAAC,IAAI,CAAC,MAAM,EAAE,CAAC;YACjB,IAAI,CAAC,MAAM,GAAG,MAAM,IAAA,yBAAY,EAAC,IAAI,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC;YACvD,MAAM,IAAA,kCAAqB,EACzB,IAAI,CAAC,MAAM,EACX,sBAAc,EACd,IAAI,CAAC,IAAI,CAAC,UAAU,CAAC,WAAW,IAAI,KAAK,CAC1C,CAAC;QACJ,CAAC;QACD,OAAO,IAAI,CAAC,MAAM,CAAC;IACrB,CAAC;IAED;;;;OAIG;IACH,KAAK,CAAC,GAAG,CAAC,IAAa;QACrB,OAAO,IAAA,oBAAQ,EACb,mBAAmB,EACnB;YACE,gBAAgB,EAAE,IAAI,CAAC,SAAS;YAChC,oBAAoB,EAAE,IAAI,CAAC,IAAI;YAC/B,0BAA0B,EAAE,IAAI,CAAC,QAAQ,EAAE,MAAM,IAAI,CAAC;SACvD,EACD,KAAK,IAAI,EAAE;YACT,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,SAAS,EAAE,CAAC;YACtC,OAAO,IAAI,CAAC,gBAAgB,CAAC,MAAM,EAAE,IAAI,CAAC,CAAC;QAC7C,CAAC,CACF,CAAC;IACJ,CAAC;IAED;;OAEG;IACH,KAAK,CAAC,OAAO,CAAC,KAAgB;QAC5B,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,SAAS,EAAE,CAAC;QACtC,MAAM,OAAO,GAAc,EAAE,CAAC;QAC9B,KAAK,MAAM,IAAI,IAAI,KAAK,EAAE,CAAC;YACzB,OAAO,CAAC,IAAI,CAAC,MAAM,IAAI,CAAC,gBAAgB,CAAC,MAAM,EAAE,IAAI,CAAC,CAAC,CAAC;QAC1D,CAAC;QACD,OAAO,OAAO,CAAC;IACjB,CAAC;IAED;;;;OAIG;IACK,KAAK,CAAC,gBAAgB,CAAC,MAAc,EAAE,IAAa;QAC1D,MAAM,MAAM,GAAG,IAAI,CAAC,IAAI,CAAC,MAAM,IAAI,OAAO,CAAC;QAC3C,MAAM,eAAe,GAAG,IAAI,CAAC,SAAS,CAAC;QACvC,MAAM,UAAU,GAAG,IAAA,iBAAS,EAAC,eAAe,EAAE,MAAM,CAAC,CAAC;QAEtD,qEAAqE;QACrE,IAAI,CAAC,IAAI,CAAC,QAAQ,IAAI,IAAI,CAAC,QAAQ,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;YACjD,MAAM,EAAE,MAAM,EAAE,GAAG,wDAAa,mBAAmB,GAAC,CAAC;YACrD,MAAM,SAAS,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;YAC7B,MAAM,IAAI,GAAG,IAAI,CAAC,IAAI,IAAI,EAAE,CAAC;YAC7B,MAAM,KAAK,GAAG,MAAM,MAAM,CACxB,MAAM,EACN,UAAU,EACV,IAAI,CAAC,IAAI,EACT,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,IAAI,CAAC,EACzB,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,EACpB,SAAS,EACT,IAAI,CAAC,KAAK,IAAI,CAAC,EACf,IAAI,CAAC,QAAQ,IAAI,CAAC,EAClB,EAAE,EACF,IAAI,CAAC,QAAQ,IAAI,CAAC,CACnB,CAAC;YACF,MAAM,GAAG,GAAG,IAAI,SAAG,CACjB,MAAM,EACN,UAAU,EACV,MAAM,CAAC,KAAK,CAAC,EACb,IAAI,CAAC,IAAI,EACT,IAAI,CAAC,IAAI,EACT,IAAI,CACL,CAAC;YACF,GAAG,CAAC,SAAS,GAAG,SAAS,CAAC;YAC1B,OAAO,EAAE,GAAG,EAAE,CAAC;QACjB,CAAC;QAED,0EAA0E;QAC1E,MAAM,YAAY,GAAyB,IAAI,GAAG,EAAE,CAAC;QACrD,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,QAAQ,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;YAC9C,MAAM,KAAK,GAAG,IAAI,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC;YAC/B,IAAI,KAAK,CAAC,QAAQ,IAAI,KAAK,CAAC,QAAQ,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;gBAChD,YAAY,CAAC,GAAG,CAAC,CAAC,EAAE,MAAM,IAAI,CAAC,gBAAgB,CAAC,MAAM,EAAE,KAAK,CAAC,CAAC,CAAC;YAClE,CAAC;QACH,CAAC;QAED,uEAAuE;QACvE,MAAM,cAAc,GAAG,IAAI,CAAC,QAAQ;aACjC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,YAAY,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC;aACtC,GAAG,CAAC,CAAC,KAAK,EAAE,EAAE;YACb,MAAM,SAAS,GAAG,KAAK,CAAC,IAAI,IAAI,EAAE,CAAC;YACnC,OAAO;gBACL,IAAI,EAAE,KAAK,CAAC,IAAI;gBAChB,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,IAAI,CAAC;gBAChC,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC,SAAS,CAAC;gBAC/B,KAAK,EAAE,SAAS,CAAC,KAAK,IAAI,CAAC;gBAC3B,QAAQ,EAAE,SAAS,CAAC,QAAQ,IAAI,CAAC;gBACjC,WAAW,EAAE,SAAS,CAAC,QAAQ,IAAI,CAAC;gBACpC,IAAI,EAAE,IAAA,iBAAS,EAAC,KAAK,CAAC,SAAS,EAAE,MAAM,CAAC;gBACxC,WAAW,EAAE,IAAA,iBAAS,EAAC,MAAM,EAAE,KAAK,CAAC,SAAS,CAAC;gBAC/C,eAAe,EAAE,eAAe;aACjC,CAAC;QACJ,CAAC,CAAC,CAAC;QAEL,uEAAuE;QACvE,MAAM,SAAS,GAAa,EAAE,CAAC;QAC/B,KAAK,MAAM,CAAC,CAAC,EAAE,OAAO,CAAC,IAAI,YAAY,CAAC,OAAO,EAAE,EAAE,CAAC;YAClD,MAAM,KAAK,GAAG,IAAI,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC;YAC/B,SAAS,CAAC,IAAI,CAAC,GAAG,IAAA,iBAAS,EAAC,MAAM,EAAE,KAAK,CAAC,SAAS,CAAC,IAAI,OAAO,CAAC,GAAG,CAAC,EAAE,EAAE,CAAC,CAAC;QAC5E,CAAC;QAED,MAAM,SAAS,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;QAC7B,MAAM,UAAU,GAAG,IAAI,CAAC,IAAI,IAAI,EAAE,CAAC;QAEnC,MAAM,GAAG,GAAG,MAAM,IAAA,eAAO,EACvB,MAAM,EACN,UAAU,EACV,IAAI,CAAC,IAAI,EACT,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,IAAI,CAAC,EACzB,IAAI,CAAC,SAAS,CAAC,UAAU,CAAC,EAC1B,SAAS,EACT,UAAU,CAAC,KAAK,IAAI,CAAC,EACrB,UAAU,CAAC,QAAQ,IAAI,CAAC,EACxB,UAAU,CAAC,QAAQ,IAAI,CAAC,EACxB,cAAc,EACd,SAAS,CACV,CAAC;QAEF,MAAM,QAAQ,GAAG,GAAG,CAAC,CAAC,CAAC,CAAC;QAExB,iEAAiE;QACjE,KAAK,MAAM,CAAC,CAAC,EAAE,OAAO,CAAC,IAAI,YAAY,CAAC,OAAO,EAAE,EAAE,CAAC;YAClD,MAAM,KAAK,GAAG,IAAI,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC;YAC/B,MAAM,SAAS,GAAG,IAAA,iBAAS,EAAC,KAAK,CAAC,SAAS,EAAE,MAAM,CAAC,CAAC;YACrD,MAAM,MAAM,CAAC,IAAI,CAAC,SAAS,CAAC,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC,EAAE;gBAC/C,QAAQ,EAAE,QAAQ;gBAClB,WAAW,EAAE,eAAe;aAC7B,CAAC,CAAC;YACH,OAAO,CAAC,GAAG,CAAC,QAAQ,GAAG,QAAQ,CAAC;YAChC,OAAO,CAAC,GAAG,CAAC,WAAW,GAAG,eAAe,CAAC;QAC5C,CAAC;QAED,kFAAkF;QAClF,MAAM,UAAU,GAAc,EAAE,CAAC;QACjC,IAAI,OAAO,GAAG,CAAC,CAAC;QAChB,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,QAAQ,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;YAC9C,IAAI,YAAY,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC;gBACxB,UAAU,CAAC,IAAI,CAAC,YAAY,CAAC,GAAG,CAAC,CAAC,CAAE,CAAC,CAAC;YACxC,CAAC;iBAAM,CAAC;gBACN,MAAM,OAAO,GAAG,GAAG,CAAC,CAAC,GAAG,OAAO,CAAC,CAAC;gBACjC,MAAM,KAAK,GAAG,IAAI,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC;gBAC/B,MAAM,QAAQ,GAAG,IAAI,SAAG,CACtB,MAAM,EACN,IAAA,iBAAS,EAAC,KAAK,CAAC,SAAS,EAAE,MAAM,CAAC,EAClC,OAAO,EACP,KAAK,CAAC,IAAI,EACV,KAAK,CAAC,IAAI,EACV,KAAK,CAAC,IAAI,IAAI,EAAE,CACjB,CAAC;gBACF,QAAQ,CAAC,SAAS,GAAG,SAAS,CAAC;gBAC/B,QAAQ,CAAC,QAAQ,GAAG,QAAQ,CAAC;gBAC7B,QAAQ,CAAC,WAAW,GAAG,eAAe,CAAC;gBACvC,UAAU,CAAC,IAAI,CAAC,EAAE,GAAG,EAAE,QAAQ,EAAE,CAAC,CAAC;gBACnC,OAAO,EAAE,CAAC;YACZ,CAAC;QACH,CAAC;QAED,MAAM,SAAS,GAAG,IAAI,SAAG,CACvB,MAAM,EACN,UAAU,EACV,QAAQ,EACR,IAAI,CAAC,IAAI,EACT,IAAI,CAAC,IAAI,EACT,UAAU,CACX,CAAC;QACF,SAAS,CAAC,SAAS,GAAG,SAAS,CAAC;QAEhC,OAAO,EAAE,GAAG,EAAE,SAAS,EAAE,QAAQ,EAAE,UAAU,EAAE,CAAC;IAClD,CAAC;IAED;;;OAGG;IACH,KAAK,CAAC,KAAK;QACT,IAAI,IAAI,CAAC,OAAO;YAAE,OAAO;QACzB,IAAI,CAAC,OAAO,GAAG,IAAI,CAAC;QACpB,IAAI,IAAI,CAAC,MAAM,EAAE,CAAC;YAChB,IAAI,CAAC,MAAM,CAAC,KAAK,EAAE,CAAC;YACpB,IAAI,CAAC,MAAM,GAAG,IAAI,CAAC;QACrB,CAAC;IACH,CAAC;CACF;AAhND,oCAgNC"}