@agentium/queue 3.1.2 → 4.0.0

package/README.md

@@ -1,36 +1,80 @@
 # @agentium/queue
 
-Background job processing for Agentium agents using BullMQ and Redis.
+Background Agent, Team and Workflow execution on BullMQ **5.81.5+ within v5 or 6.3.11+ within v6** and Redis. Node 22.18+ or 24.11+ is required. New queues can use v6; migrate persisted legacy repeat records with v5 before deploying v6 workers or writers.
 
-## Install
+```sh
+npm install @agentium/queue bullmq@^6.3.11 ioredis
+```
+
+```ts
+import { AgentQueue, AgentWorker } from "@agentium/queue";
 
-```bash
-npm install @agentium/queue bullmq ioredis
+const connection = "redis://127.0.0.1:6379/0";
+const queue = new AgentQueue({ connection, defaultJobOptions: {
+  attempts: 3, backoff: { type: "exponential", delay: 1000 },
+} });
+const worker = new AgentWorker({ connection, agentRegistry: { assistant: agent } });
+await queue.enqueueAgentRun({ agentName: "assistant", input: "Summarize the report" });
+await queue.schedule({ id: "daily-summary", cron: "0 9 * * *", timezone: "UTC",
+  agent: { name: "assistant", input: "Summarize new reports" },
+});
+// The same ID updates its schedule; it does not create another scheduler.
+await queue.unschedule("daily-summary");
+await worker.stop();
+await queue.close();
 ```
 
-## Quick Start
+The default queue name is now `agentium-jobs`; the old `agentium:jobs` was rejected by current BullMQ because queue names cannot contain colons. Set the same explicit, valid `queueName` on producer and worker to continue an existing deployment. Constructors load BullMQ through Node's ESM-compatible `createRequire`. Redis URLs preserve credentials, database and `rediss:` TLS settings.
 
-```typescript
-import { Agent, openai } from "@agentium/core";
-import { AgentQueue, AgentWorker } from "@agentium/queue";
+`schedule` accepts exactly one `agent`, `team` or `workflow` target, and uses a stable caller-supplied Job Scheduler ID. Workflow payloads forward `initialState`, `sessionId`, `userId` and `tenantId` into `Workflow.run`. Agents and Teams also forward identity. Queue access is a trusted host boundary: authenticate producers and authorize these claims before enqueueing; a Redis payload is not proof of identity.
+
+Retry configuration belongs to the producer's `defaultJobOptions` or individual enqueue options. The previous `WorkerConfig.attempts/backoffDelay` fields never controlled BullMQ jobs; they have been removed from the TypeScript API. JavaScript callers still receive a migration diagnostic before BullMQ or a connection is initialized. Retrying effectful agent work requires application idempotency. `cancelJob` removes a waiting/delayed job and cannot cancel an active locked job. `stop` drains active work and reports its timeout; it does not claim that remote effects were rolled back.
+
+## Persisted repeat schedule migration
 
-const agent = new Agent({ name: "assistant", model: openai("gpt-4o") });
-const connection = { host: "localhost", port: 6379 };
+BullMQ v6 removes legacy repeat APIs. `listLegacySchedules()` and `removeLegacySchedule()` require a BullMQ v5 installation and fail clearly on v6, including on clean queues. On v6, `schedule`, recurring enqueue and `listSchedules` refuse a queue containing legacy metadata; they do not migrate it or silently hide it. See the [official v5-to-v6 migration guide](https://docs.bullmq.io/guide/migrations/migrate-from-v5-to-v6).
 
-const queue = new AgentQueue({ connection });
-const worker = new AgentWorker({ agents: { assistant: agent }, connection });
+Perform the following maintenance with BullMQ 5.81.5 (or a compatible v5 release). Legacy records stay in Redis until an operator removes them. Do not run old and new schedule writers concurrently.
 
-await queue.enqueueAgentRun({ agentName: "assistant", input: "Hello!" });
+1. Stop schedule writers, call `queue.pause()`, let active jobs finish, then stop workers. `removeLegacySchedule` requires both a paused queue and zero active jobs.
+2. Save `await queue.listLegacySchedules()` and the original application payloads/options. The inventory includes keys, names, cron, timezone and next timestamp; it is not a complete payload backup. Back up Redis before changes.
+3. Choose stable new IDs and deduplicate the desired schedule set. Remove each legacy key with `removeLegacySchedule(key)` before calling `schedule`. New creation refuses a matching legacy name/cron to prevent accidental overlap.
+4. Verify `listLegacySchedules()` is empty for the whole queue and `listSchedules()` contains exactly the intended IDs. Close the v5 maintenance client. Deploy v6 workers and writers, verify `listSchedules()` again, then `await queue.resume()` and enable schedule writers.
+5. To roll back, stop v6 workers/writers and return to v5 while paused and drained. Remove replacement IDs using `unschedule`, restore the old records with saved original BullMQ repeat options and payloads (or restore the Redis backup), verify the inventory, and restart the old deployment. Never restore legacy records while replacements are active.
+
+The convenience `enqueue*({repeat})` APIs now upsert schedulers using a hash of target, payload and repeat options. Changing the payload changes that hash; use `schedule({id})` when you need stable updates. `listSchedules` excludes legacy records on v5 and rejects them on v6. BullMQ stores both formats in the repeat index: the adapter recognizes new schedulers by their returned `iterationCount` and also handles the v6 SDK diagnostic for older legacy keys. The migration check is not a lock against concurrent legacy writers; stopping them remains required.
+
+## Verification
+
+Run this command from the repository root after building core:
+
+```sh
+AGENTIUM_REDIS_TEST=1 AGENTIUM_REDIS_PORT=6389 npx vitest run packages/queue/src/__tests__
 ```
 
-## Documentation
+Use a disposable Redis database. Fixtures create unique queues and remove only those queues. CI runs them against an isolated Redis 7 service. They exercise the real BullMQ 5.81.5 and 6.3.11 SDKs: scheduler upserts/removal, v5 persisted legacy data rejected by v6, explicit paused v5 migration, v6 use of migrated scheduler records, older legacy-key rejection, worker identity/progress isolation and durable delivery. The fixture uses the `bullmq-v5` development alias for the v5 maintenance process; it does not emulate SDK methods. Redis Cluster and alternative BullMQ backends are outside this fixture.
+
+## Opt-in durable delivery
 
-Full docs at [docs.agentium.in](https://docs.agentium.in)
+`DurableAgentQueue` and `DurableAgentWorker` use a separate `agentium-durable` queue. The task store owns execution state; Redis delivers a versioned hint containing only tenant/task IDs and an immutable definition digest. Existing Agent/Team/Workflow workers keep their current behavior.
 
-## Community
+```ts
+import { DurableAgentQueue, DurableAgentWorker } from '@agentium/queue';
+// supervisor uses the same initialized durable store on all workers.
+const durable = new DurableAgentQueue({ connection, supervisor, admit: admitTask });
+const durableWorker = new DurableAgentWorker({
+  connection, supervisor, admit: recheckCurrentGrants,
+  drivers: [{ id: 'save-document', version: 1, recoverable: true, execute: saveDocument }],
+});
+await durable.enqueue({
+  id: taskId, identity: verifiedIdentity, manifestHash, inputRef,
+  policyRevision: 1, grantRefs: approvedGrants,
+  driver: { id: 'save-document', version: 1 }, input: { documentRef },
+});
+```
 
-Join the conversation on [Discord](https://discord.gg/T86SJshP).
+Both callbacks are mandatory trusted-host admission boundaries. Producer admission happens before persistence; worker admission happens before and after the atomic claim. Versioned drivers must explicitly support replay with stable action IDs and reconciliation. Registering an ordinary Agent/Team/Workflow function does not establish that property.
 
-## License
+Persistence precedes Redis enqueue. If enqueue fails, the task remains recoverable: retry the same immutable definition or call `wake({tenantId, taskId})`. Wake explicitly after approval or reconciliation. Duplicate delivery is fenced by the store; an early retry remains delayed until an existing lease can be reclaimed. Completed task deliveries return only task/state/revision metadata to Redis. Unknown external outcomes block handler execution until reconciliation.
 
-MIT
+`cancel` persists cancellation and wakes a worker; it does not claim immediate remote termination. The host must retry `wake` when Redis is unavailable. `stop` drains active work. Maintain a host recovery sweep for persisted work without a delivery hint; this package does not scan arbitrary stores or start hidden background services. See the [durable store and recovery contract](../core/src/durable/README.md).

package/dist/connection.d.ts

@@ -0,0 +1,27 @@
+export declare function queueConnection(value: string | {
+    host: string;
+    port: number;
+    password?: string;
+    db?: number;
+    tls?: boolean;
+}): {
+    tls: {};
+    host: string;
+    port: number;
+    password?: string;
+    db?: number;
+} | {
+    tls: undefined;
+    host: string;
+    port: number;
+    password?: string;
+    db?: number;
+} | {
+    tls?: {} | undefined;
+    password?: string | undefined;
+    username?: string | undefined;
+    host: string;
+    port: number;
+    db: number;
+};
+//# sourceMappingURL=connection.d.ts.map

package/dist/connection.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"connection.d.ts","sourceRoot":"","sources":["../src/connection.ts"],"names":[],"mappings":"AAAA,wBAAgB,eAAe,CAC7B,KAAK,EAAE,MAAM,GAAG;IAAE,IAAI,EAAE,MAAM,CAAC;IAAC,IAAI,EAAE,MAAM,CAAC;IAAC,QAAQ,CAAC,EAAE,MAAM,CAAC;IAAC,EAAE,CAAC,EAAE,MAAM,CAAC;IAAC,GAAG,CAAC,EAAE,OAAO,CAAA;CAAE;;UAArE,MAAM;UAAQ,MAAM;eAAa,MAAM;SAAO,MAAM;;;UAApD,MAAM;UAAQ,MAAM;eAAa,MAAM;SAAO,MAAM;;;;;;;;EAe7E"}

package/dist/durable.d.ts

@@ -0,0 +1,58 @@
+import { type DurableTaskHandler, type DurableTaskInput, type DurableTaskKey, type DurableTaskRecord, type DurableTaskSupervisor } from "@agentium/core";
+import type { QueueConfig } from "./job-producer.js";
+export interface DurableJobEnvelope {
+    schemaVersion: 1;
+    type: "durable";
+    tenantId: string;
+    taskId: string;
+    inputDigest: string;
+}
+export interface DurableDriverRegistration {
+    id: string;
+    version: number;
+    /** Explicit host assertion: resumes use stable action IDs and reconcile uncertain effects. */
+    recoverable: true;
+    execute: DurableTaskHandler;
+}
+export interface DurableQueueConfig extends QueueConfig {
+    supervisor: DurableTaskSupervisor;
+    /** Verify identity, definition, policy and grants before writing any task/queue record. */
+    admit: (task: Readonly<DurableTaskInput>) => Promise<void>;
+}
+/** Queue delivery is a hint; Mongo/another atomic task store owns authoritative state.
+ * Persistence precedes enqueue. If Redis fails, call wake() after retrying admission.
+ * The stable task key prevents a delivery retry from creating a second execution.
+ */
+export declare class DurableAgentQueue {
+    private readonly config;
+    private queue;
+    constructor(config: DurableQueueConfig);
+    enqueue(task: DurableTaskInput): Promise<{
+        taskId: string;
+        deliveryId: string;
+    }>;
+    /** Explicit re-delivery after approval, reconciliation, or a failed initial enqueue. */
+    wake(key: DurableTaskKey): Promise<{
+        taskId: string;
+        deliveryId: string;
+    }>;
+    cancel(key: DurableTaskKey, reason?: string): Promise<void>;
+    close(): Promise<void>;
+}
+export interface DurableWorkerConfig extends Omit<QueueConfig, "defaultJobOptions"> {
+    supervisor: DurableTaskSupervisor;
+    workerId?: string;
+    concurrency?: number;
+    drivers: readonly DurableDriverRegistration[];
+    /** Revalidate policy/grants from authoritative host state; a payload is not authentication. */
+    admit: (task: Readonly<DurableTaskRecord>) => Promise<void>;
+    /** Observe delivery failures; durable task outcome remains available from the store. */
+    onError?: (error: Error) => void;
+}
+export declare class DurableAgentWorker {
+    private worker;
+    constructor(config: DurableWorkerConfig);
+    /** Drains owned work. Active durable cancellation is explicit through queue.cancel(). */
+    stop(): Promise<void>;
+}
+//# sourceMappingURL=durable.d.ts.map

package/dist/durable.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"durable.d.ts","sourceRoot":"","sources":["../src/durable.ts"],"names":[],"mappings":"AAEA,OAAO,EAEL,KAAK,kBAAkB,EACvB,KAAK,gBAAgB,EACrB,KAAK,cAAc,EACnB,KAAK,iBAAiB,EACtB,KAAK,qBAAqB,EAG3B,MAAM,gBAAgB,CAAC;AAExB,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,mBAAmB,CAAC;AAGrD,MAAM,WAAW,kBAAkB;IACjC,aAAa,EAAE,CAAC,CAAC;IACjB,IAAI,EAAE,SAAS,CAAC;IAChB,QAAQ,EAAE,MAAM,CAAC;IACjB,MAAM,EAAE,MAAM,CAAC;IACf,WAAW,EAAE,MAAM,CAAC;CACrB;AACD,MAAM,WAAW,yBAAyB;IACxC,EAAE,EAAE,MAAM,CAAC;IACX,OAAO,EAAE,MAAM,CAAC;IAChB,8FAA8F;IAC9F,WAAW,EAAE,IAAI,CAAC;IAClB,OAAO,EAAE,kBAAkB,CAAC;CAC7B;AACD,MAAM,WAAW,kBAAmB,SAAQ,WAAW;IACrD,UAAU,EAAE,qBAAqB,CAAC;IAClC,2FAA2F;IAC3F,KAAK,EAAE,CAAC,IAAI,EAAE,QAAQ,CAAC,gBAAgB,CAAC,KAAK,OAAO,CAAC,IAAI,CAAC,CAAC;CAC5D;AA6BD;;;GAGG;AACH,qBAAa,iBAAiB;IAEhB,OAAO,CAAC,QAAQ,CAAC,MAAM;IADnC,OAAO,CAAC,KAAK,CAAM;gBACU,MAAM,EAAE,kBAAkB;IAOjD,OAAO,CAAC,IAAI,EAAE,gBAAgB,GAAG,OAAO,CAAC;QAAE,MAAM,EAAE,MAAM,CAAC;QAAC,UAAU,EAAE,MAAM,CAAA;KAAE,CAAC;IAoBtF,wFAAwF;IAClF,IAAI,CAAC,GAAG,EAAE,cAAc,GAAG,OAAO,CAAC;QAAE,MAAM,EAAE,MAAM,CAAC;QAAC,UAAU,EAAE,MAAM,CAAA;KAAE,CAAC;IAO1E,MAAM,CAAC,GAAG,EAAE,cAAc,EAAE,MAAM,CAAC,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAO3D,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC;CAG7B;AAED,MAAM,WAAW,mBAAoB,SAAQ,IAAI,CAAC,WAAW,EAAE,mBAAmB,CAAC;IACjF,UAAU,EAAE,qBAAqB,CAAC;IAClC,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,OAAO,EAAE,SAAS,yBAAyB,EAAE,CAAC;IAC9C,+FAA+F;IAC/F,KAAK,EAAE,CAAC,IAAI,EAAE,QAAQ,CAAC,iBAAiB,CAAC,KAAK,OAAO,CAAC,IAAI,CAAC,CAAC;IAC5D,wFAAwF;IACxF,OAAO,CAAC,EAAE,CAAC,KAAK,EAAE,KAAK,KAAK,IAAI,CAAC;CAClC;AACD,qBAAa,kBAAkB;IAC7B,OAAO,CAAC,MAAM,CAAM;gBACR,MAAM,EAAE,mBAAmB;IAoDvC,yFAAyF;IACnF,IAAI,IAAI,OAAO,CAAC,IAAI,CAAC;CAG5B"}