@cogitator-ai/worker 0.9.24 → 0.11.0

package/README.md

@@ -8,7 +8,7 @@ Distributed job queue for Cogitator agent execution. Built on BullMQ for reliabl
 pnpm add @cogitator-ai/worker @cogitator-ai/core ioredis
 ```
 
-BullMQ and `@cogitator-ai/swarms` come as dependencies; `ioredis` is a peer dependency. Website docs: [Worker Queues](https://cogitator.app/docs/deployment/worker-queues), [Distributed Swarms](https://cogitator.app/docs/swarms/distributed).
+BullMQ, `@cogitator-ai/swarms` and `@cogitator-ai/workflows` come as dependencies, `ioredis` is a peer dependency. Website docs: [Worker Queues](https://cogitator.app/docs/deployment/worker-queues), [Distributed Swarms](https://cogitator.app/docs/swarms/distributed).
 
 ## Features
 
@@ -16,7 +16,9 @@ BullMQ and `@cogitator-ai/swarms` come as dependencies; `ioredis` is a peer depe
 - **Job Types** - Agents, workflow graphs, swarms and distributed swarm turns
 - **Worker Runtime** - Run jobs with your own `Cogitator` (provider keys, memory) and tool implementations
 - **Distributed Swarms** - `DistributedSwarmWorker` executes agent turns for `@cogitator-ai/swarms`
-- **Auto-Retry** - Exponential backoff for failed jobs
+- **Auto-Retry** - Exponential backoff for failed jobs, a retried workflow job goes on from its last checkpoint
+- **Job Ids** - Adding a job under an id the queue holds gives that job back, so producers can retry an add
+- **Usage** - Workflow and swarm results carry tokens and cost, the metrics count them by job type
 - **Priority Queue** - Process important jobs first
 - **Delayed Jobs** - Schedule jobs for later execution
 - **Prometheus Metrics** - Built-in HPA support
@@ -129,6 +131,8 @@ interface QueueConfig {
 }
 ```
 
+`redis` is the `RedisConnectionConfig` of `@cogitator-ai/types`, the same config `DistributedSwarmWorker` and distributed swarms take. `resolveRedisOptions(redis)` gives the settings it resolves to for one server.
+
 ### Adding Jobs
 
 **Agent Jobs:**
@@ -188,7 +192,7 @@ The worker routes `model` exactly like the same agent in-process: a prefix that
 
 **Workflow Jobs:**
 
-Workflow jobs run a DAG over a shared state object initialised from the job input. Nodes run as soon as their predecessors settle; independent branches run concurrently.
+Workflow jobs run on the `WorkflowExecutor` of `@cogitator-ai/workflows`, over a shared state object initialised from the job input. Nodes run in rounds, each running at once every node whose nodes before it all settled, and a graph with many roots starts at all of them.
 
 ```typescript
 const workflowConfig: SerializedWorkflow = {
@@ -233,7 +237,9 @@ await queue.addWorkflowJob(workflowConfig, { ticket: 'App crashes on login' });
 | `condition` | `{ key, operator, value? }` — `equals`, `not-equals`, `contains`, `exists`, `gt`, `lt`; outgoing edges use `condition: 'true' \| 'false'` |
 | `parallel`  | `{}` — fan-out marker; successors run concurrently                                                                                        |
 
-Node outputs are stored in the state under `outputKey` (default: node id). Nodes reachable only through untaken condition branches are skipped (`{ skipped: true }` in `nodeResults`). Graphs are validated before execution (unknown nodes, invalid configs, cycles).
+Node outputs are stored in the state under `outputKey` (default: node id). Nodes reachable only through untaken condition branches are skipped (`{ skipped: true }` in `nodeResults`). Graphs are validated before execution (unknown nodes, invalid configs, cycles). The run keeps its own bookkeeping (which nodes ran, the branch a condition took, the usage of each agent node) in state keys starting with `__cogitator:`, which are left out of the job's `output`. Input keys, node ids and output keys starting with `__cogitator:` are refused.
+
+A workflow job keeps one checkpoint, saved over after every node under the job's id and `runId`, so jobs sharing a `runId` never resume from each other. A job BullMQ retries (`attempts`) goes on from it instead of running its finished agent nodes again. The checkpoint is deleted once the run finished, and once no attempt follows a failure: after the last attempt, a cancelled job or a paused agent run. A job a shutdown stopped keeps it for the worker that takes the job up. `WorkerPool` keeps the checkpoints in its Redis under `<queue prefix>:<queue name>:workflow-checkpoints` (`cogitator`, or `{cogitator}` in cluster mode) for `checkpointTtl` seconds after the last save (7 days by default), so the checkpoint of a run whose worker crashed goes on its own. `checkpoints` in the worker config takes a `CheckpointStore` of your own.
 
 **Swarm Jobs:**
 
@@ -260,6 +266,18 @@ await queue.addSwarmJob(swarmConfig, 'Write an article about AI', {
 | `debate`        | `debate`       | `maxRounds` rounds, `coordinator` moderates                                                                                                                                    |
 | `voting`        | `consensus`    | `consensusThreshold`, `maxRounds`, `coordinator` breaks ties                                                                                                                   |
 
+**Job Ids:**
+
+`addAgentJob`, `resumeAgentJob`, `addWorkflowJob`, `addSwarmJob` and `addSwarmAgentJob` take a `jobId`, and the job is added under it. Adding a job whose id the queue holds already gives that job back instead of adding a second one, so a producer can retry an add, such as after a timeout, without running the work twice:
+
+```typescript
+const job = await queue.addWorkflowJob(workflowConfig, input, { jobId: `triage-${ticketId}` });
+const again = await queue.addWorkflowJob(workflowConfig, input, { jobId: `triage-${ticketId}` });
+// again.id === job.id, one job runs
+```
+
+Two adds racing with the same id both get the job the queue holds. A job the queue removed (past `removeOnComplete` or `removeOnFail`) is gone, and its id adds a new job. BullMQ builds Redis keys from the id, so an empty id, one with `:`, an integer and a name of the queue's own keys (`meta`, `id`, `events`, `stalled-check` and the like) are refused, and an id the queue holds as another kind of job (an agent job for `addWorkflowJob`, say) throws. Without `jobId` every add gets a new id.
+
 ### Queue Methods
 
 ```typescript
@@ -335,6 +353,8 @@ interface WorkerConfig extends QueueConfig {
   stalledInterval?: number; // Default: 30000ms
   cogitator?: Cogitator; // Default: new Cogitator()
   tools?: Tool[]; // Tool implementations, resolved by name
+  checkpoints?: CheckpointStore; // Where workflow jobs keep their checkpoints. Default: the pool's Redis
+  checkpointTtl?: number; // Seconds the default store keeps a checkpoint after its last save. Default: 7 days
 }
 ```
 
@@ -391,7 +411,7 @@ Built-in processors handle each job type.
 
 ### Using Processors Directly
 
-Processors take the job payload and an optional runtime (`{ cogitator, tools }`):
+Processors take the job payload and an optional runtime (`{ cogitator, tools, checkpoints }`):
 
 ```typescript
 import { processAgentJob, processWorkflowJob, processSwarmJob } from '@cogitator-ai/worker';
@@ -414,7 +434,9 @@ const swarmResult = await processSwarmJob(
 );
 ```
 
-`processSwarmAgentJob(payload, { publisher, isFinalAttempt, ...runtime })` executes one distributed swarm turn and publishes the result (tagged with the job id) to `payload.stateKeys.results`; `executeSwarmAgentJob` returns the result without publishing.
+`processWorkflowJob` called on its own runs without checkpoints unless the runtime has `checkpoints` (a `CheckpointStore`). With them, pass `{ willRetry }` as the third argument: a run whose failure another attempt follows (`willRetry(error)` returns `true`) keeps its checkpoint for it, any other failure deletes it.
+
+`processSwarmAgentJob(payload, { publisher, isFinalAttempt, ...runtime })` executes one distributed swarm turn and hands the result (tagged with the job id) to the coordinator with `deliverSwarmJobResult` of `@cogitator-ai/swarms`: when the job names `stateKeys.result` and the publisher has `set`, the result is kept there for `payload.resultTtl` ms (one hour by default), then it is published to `payload.stateKeys.results`. `publisher` is a `SwarmResultPublisher`, now an alias of `SwarmResultClient` (`publish`, optional `set`). `executeSwarmAgentJob` returns the result without publishing.
 
 ---
 
@@ -428,7 +450,7 @@ import { DistributedSwarmWorker } from '@cogitator-ai/worker';
 
 const worker = new DistributedSwarmWorker(
   {
-    redis: { host: 'localhost', port: 6379 },
+    redis: { url: 'redis://localhost:6379' }, // the same Redis as the swarm's distributed.redis
     keyPrefix: 'swarm', // must match the swarm's distributed.redis.keyPrefix
     queue: 'swarm-agent-jobs', // must match distributed.queue
     concurrency: 4,
@@ -439,6 +461,7 @@ const worker = new DistributedSwarmWorker(
     onJobCompleted: (job) => console.log('done', job.agentName),
     onJobFailed: (job, error) => console.error(job.agentName, error.message),
     onJobCancelled: (job) => console.log('the swarm gave up on', job.jobId),
+    onJobsRecovered: (workerId, count) => console.warn(`${count} turns of ${workerId} requeued`),
     onError: (error) => console.error(error),
   }
 );
@@ -449,6 +472,18 @@ process.on('SIGTERM', () => void worker.stop()); // waits for in-flight turns
 
 Failed turns are reported back to the swarm as errors, so the swarm's own `errorHandling` (retry, failover, skip) applies. A turn the swarm gave up on (its run aborted or timed out, the swarm closed, another attempt answered) is skipped, or aborted while it runs (checked every `cancelCheckInterval` ms, default 1000), and reported to `onJobCancelled` instead of being published. Each turn runs on the model its agent would use in-process, routed by the worker's `cogitator`, so give the worker the same `llm.backends`, plugins and provider keys as the process that runs the swarm.
 
+`redis` takes a `url` (`rediss://` for TLS), `host`, `port`, `username`, `password`, `db`, `tls` or `cluster.nodes`, like the queue. The turns sit in the list `<keyPrefix>:jobs:{<queue>}`, whose braces are a Redis Cluster hash tag that keeps the queue and its worker keys in one slot. Earlier versions used `<keyPrefix>:jobs:<queue>`, so upgrade the processes that run swarms and the workers together.
+
+The queue is reliable:
+
+- A worker moves each turn with `BLMOVE` (Redis 6.2 or newer) from the queue to its own list `<queue key>:processing:<workerId>`, and takes it off once the turn is done.
+- Every `heartbeatInterval` ms (default 5000) it sets its heartbeat key `<queue key>:heartbeat:<workerId>`, which expires after three intervals, and stays in the set `<queue key>:workers`.
+- On `start()` and on every beat, a worker moves the turns of workers whose heartbeat is gone back to the head of the queue, drops those workers from the set and reports `onJobsRecovered(workerId, count)`.
+- A worker also checks its own list on every beat: a turn on it that the worker does not run at two beats in a row, such as one a `BLMOVE` moved while a dropped connection lost its reply, goes back to the head of the queue and is reported with the worker's own id.
+- A worker silent for three intervals counts as gone, so a turn on a stalled worker can run twice.
+- The result is kept under the job's `stateKeys.result` for `resultTtl` ms before it is published, so the coordinator finds it after a missed message, and a worker skips a job whose result is there (such as one a dead worker answered before it took the turn off its list). A result the worker cannot keep under its key is still published.
+- `stop()` waits for in-flight turns, hands back what is left on its list, leaves the set and deletes its heartbeat. `getWorkerId()` gives the worker's id, new on every `start()`.
+
 ---
 
 ## Job Results
@@ -508,8 +543,17 @@ interface WorkflowJobResult {
   type: 'workflow';
   output: Record<string, unknown>;
   nodeResults: Record<string, unknown>;
+  usage: JobUsage; // what the agent nodes used and cost
   duration: number;
 }
+
+interface JobUsage {
+  inputTokens?: number; // absent where the runs reported only their totals
+  outputTokens?: number;
+  totalTokens: number;
+  cost: number; // USD
+  runs: number; // agent runs counted
+}
 ```
 
 ### Swarm Job Result
@@ -519,6 +563,8 @@ interface SwarmJobResult {
   type: 'swarm';
   output: string;
   rounds: number;
+  // what the swarm's agent runs used and cost, in all and by agent
+  usage: JobUsage & { byAgent: Record<string, { tokens: number; cost: number; runs: number }> };
   agentOutputs: {
     agent: string;
     output: string;
@@ -526,6 +572,8 @@ interface SwarmJobResult {
 }
 ```
 
+A swarm reports only total tokens, so its `usage` has no `inputTokens` or `outputTokens`.
+
 ---
 
 ## Prometheus Metrics
@@ -554,20 +602,24 @@ app.listen(9090);
 
 ### Available Metrics
 
-| Metric                           | Type      | Description                                                 |
-| -------------------------------- | --------- | ----------------------------------------------------------- |
-| `cogitator_queue_depth`          | gauge     | Total waiting + delayed jobs                                |
-| `cogitator_queue_waiting`        | gauge     | Jobs ready to run, prioritized jobs included                |
-| `cogitator_queue_active`         | gauge     | Jobs currently being processed                              |
-| `cogitator_queue_completed`      | gauge     | Completed jobs kept in Redis (capped by `removeOnComplete`) |
-| `cogitator_queue_failed`         | gauge     | Failed jobs kept in Redis (capped by `removeOnFail`)        |
-| `cogitator_queue_delayed`        | gauge     | Scheduled/delayed jobs                                      |
-| `cogitator_workers_total`        | gauge     | Workers connected to the queue                              |
-| `cogitator_job_duration_seconds` | histogram | Job processing time                                         |
-| `cogitator_jobs_by_type_total`   | counter   | Jobs by type                                                |
-| `cogitator_jobs_failed_total`    | counter   | Jobs that failed their last attempt, by `type`              |
-
-`cogitator_queue_completed` and `cogitator_queue_failed` can go down as BullMQ trims old jobs, so alert on `cogitator_jobs_failed_total` instead (for example `increase(cogitator_jobs_failed_total[5m]) > 5`). The per-type counters appear after the first job of that kind.
+| Metric                              | Type      | Description                                                 |
+| ----------------------------------- | --------- | ----------------------------------------------------------- |
+| `cogitator_queue_depth`             | gauge     | Total waiting + delayed jobs                                |
+| `cogitator_queue_waiting`           | gauge     | Jobs ready to run, prioritized jobs included                |
+| `cogitator_queue_active`            | gauge     | Jobs currently being processed                              |
+| `cogitator_queue_completed`         | gauge     | Completed jobs kept in Redis (capped by `removeOnComplete`) |
+| `cogitator_queue_failed`            | gauge     | Failed jobs kept in Redis (capped by `removeOnFail`)        |
+| `cogitator_queue_delayed`           | gauge     | Scheduled/delayed jobs                                      |
+| `cogitator_workers_total`           | gauge     | Workers connected to the queue                              |
+| `cogitator_job_duration_seconds`    | histogram | Job processing time                                         |
+| `cogitator_jobs_by_type_total`      | counter   | Jobs by type                                                |
+| `cogitator_jobs_failed_total`       | counter   | Jobs that failed their last attempt, by `type`              |
+| `cogitator_job_tokens_total`        | counter   | Tokens the agent runs of completed jobs used, by `type`     |
+| `cogitator_job_input_tokens_total`  | counter   | Input tokens of completed jobs, by `type`                   |
+| `cogitator_job_output_tokens_total` | counter   | Output tokens of completed jobs, by `type`                  |
+| `cogitator_job_cost_usd_total`      | counter   | USD the agent runs of completed jobs cost, by `type`        |
+
+`cogitator_queue_completed` and `cogitator_queue_failed` can go down as BullMQ trims old jobs, so alert on `cogitator_jobs_failed_total` instead (for example `increase(cogitator_jobs_failed_total[5m]) > 5`). The per-type counters appear after the first job of that kind. The usage counters take the `usage` of completed agent, workflow, swarm and swarm-agent jobs, input and output tokens only where the job reported them apart (swarm jobs do not).
 
 ### Duration Histogram
 
@@ -587,7 +639,7 @@ histogram.reset();
 
 ### Metrics Collector
 
-`WorkerPool` keeps one in `pool.metrics`: it calls `recordJob(type, durationMs)` for each completed job and `recordFailure(type)` for each job that failed its last attempt (retried attempts are not counted). Use your own collector when you process jobs outside the pool.
+`WorkerPool` keeps one in `pool.metrics`: it calls `recordJob(type, durationMs)` and `recordUsage(type, result.usage)` for each completed job and `recordFailure(type)` for each job that failed its last attempt (retried attempts are not counted). Use your own collector when you process jobs outside the pool.
 
 ```typescript
 import { MetricsCollector } from '@cogitator-ai/worker';
@@ -596,6 +648,7 @@ const collector = new MetricsCollector();
 
 collector.recordJob('agent', 1500);
 collector.recordJob('workflow', 3200);
+collector.recordUsage('workflow', workflowResult.usage); // { inputTokens?, outputTokens?, totalTokens, cost }
 collector.recordFailure('agent');
 
 const output = collector.format(queueMetrics, { queue: 'main' });
@@ -885,6 +938,7 @@ import type {
   AgentJobResult,
   WorkflowJobResult,
   SwarmJobResult,
+  JobUsage,
   SwarmAgentJobResult,
 
   // Configuration

package/dist/connection.d.ts

@@ -1,23 +1,8 @@
 import { Cluster, Redis } from 'ioredis';
 import type { ConnectionOptions } from 'bullmq';
 import type { QueueConfig } from './types.js';
+export { resolveRedisOptions, type RedisConnectionOptions } from '@cogitator-ai/swarms';
 export declare const DEFAULT_QUEUE_NAME = "cogitator-jobs";
-/** A single Redis server's connection settings, the url read and the explicit fields applied. */
-export interface RedisConnectionOptions {
-    host: string;
-    port: number;
-    username?: string;
-    password?: string;
-    db?: number;
-    tls?: Record<string, never>;
-}
-/**
- * Connection settings from the queue's redis config: `url` first (`redis://` or `rediss://` for
- * TLS, `user:password@`, a `/db` path), then the explicit fields, which win.
- *
- * @throws Error for a url that is not redis:// or rediss://, or a db path that is not a number
- */
-export declare function resolveRedisOptions(redis: RedisConnectionConfig): RedisConnectionOptions;
 export type RedisClient = Redis | Cluster;
 type RedisConnectionConfig = QueueConfig['redis'];
 /**
@@ -41,5 +26,4 @@ export declare function createBullConnection(redis: RedisConnectionConfig, optio
  * Plain client (e.g. for publishing swarm results) using the same Redis configuration.
  */
 export declare function createRedisClient(redis: RedisConnectionConfig): RedisClient;
-export {};
 //# sourceMappingURL=connection.d.ts.map

package/dist/connection.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"connection.d.ts","sourceRoot":"","sources":["../src/connection.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,OAAO,EAAE,KAAK,EAAE,MAAM,SAAS,CAAC;AACzC,OAAO,KAAK,EAAE,iBAAiB,EAAE,MAAM,QAAQ,CAAC;AAChD,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,SAAS,CAAC;AAE3C,eAAO,MAAM,kBAAkB,mBAAmB,CAAC;AAEnD,iGAAiG;AACjG,MAAM,WAAW,sBAAsB;IACrC,IAAI,EAAE,MAAM,CAAC;IACb,IAAI,EAAE,MAAM,CAAC;IACb,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,EAAE,CAAC,EAAE,MAAM,CAAC;IACZ,GAAG,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,KAAK,CAAC,CAAC;CAC7B;AAED;;;;;GAKG;AACH,wBAAgB,mBAAmB,CAAC,KAAK,EAAE,qBAAqB,GAAG,sBAAsB,CAgCxF;AAED,MAAM,MAAM,WAAW,GAAG,KAAK,GAAG,OAAO,CAAC;AAE1C,KAAK,qBAAqB,GAAG,WAAW,CAAC,OAAO,CAAC,CAAC;AAElD;;GAEG;AACH,wBAAgB,WAAW,CAAC,KAAK,EAAE,qBAAqB,GAAG,MAAM,CAEhE;AAED,MAAM,WAAW,cAAc;IAC7B,UAAU,EAAE,iBAAiB,CAAC;IAC9B,yEAAyE;IACzE,OAAO,IAAI,OAAO,CAAC,IAAI,CAAC,CAAC;CAC1B;AAaD;;;;GAIG;AACH,wBAAgB,oBAAoB,CAClC,KAAK,EAAE,qBAAqB,EAC5B,OAAO,GAAE;IAAE,QAAQ,CAAC,EAAE,OAAO,CAAA;CAAO,GACnC,cAAc,CA4BhB;AAED;;GAEG;AACH,wBAAgB,iBAAiB,CAAC,KAAK,EAAE,qBAAqB,GAAG,WAAW,CAS3E"}
+{"version":3,"file":"connection.d.ts","sourceRoot":"","sources":["../src/connection.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,OAAO,EAAE,KAAK,EAAE,MAAM,SAAS,CAAC;AACzC,OAAO,KAAK,EAAE,iBAAiB,EAAE,MAAM,QAAQ,CAAC;AAEhD,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,SAAS,CAAC;AAE3C,OAAO,EAAE,mBAAmB,EAAE,KAAK,sBAAsB,EAAE,MAAM,sBAAsB,CAAC;AAExF,eAAO,MAAM,kBAAkB,mBAAmB,CAAC;AAEnD,MAAM,MAAM,WAAW,GAAG,KAAK,GAAG,OAAO,CAAC;AAE1C,KAAK,qBAAqB,GAAG,WAAW,CAAC,OAAO,CAAC,CAAC;AAElD;;GAEG;AACH,wBAAgB,WAAW,CAAC,KAAK,EAAE,qBAAqB,GAAG,MAAM,CAEhE;AAED,MAAM,WAAW,cAAc;IAC7B,UAAU,EAAE,iBAAiB,CAAC;IAC9B,yEAAyE;IACzE,OAAO,IAAI,OAAO,CAAC,IAAI,CAAC,CAAC;CAC1B;AAED;;;;GAIG;AACH,wBAAgB,oBAAoB,CAClC,KAAK,EAAE,qBAAqB,EAC5B,OAAO,GAAE;IAAE,QAAQ,CAAC,EAAE,OAAO,CAAA;CAAO,GACnC,cAAc,CA4BhB;AAED;;GAEG;AACH,wBAAgB,iBAAiB,CAAC,KAAK,EAAE,qBAAqB,GAAG,WAAW,CAS3E"}

package/dist/connection.js

@@ -1,58 +1,13 @@
 import { Cluster, Redis } from 'ioredis';
+import { clusterNodeOptions, resolveRedisOptions } from '@cogitator-ai/swarms';
+export { resolveRedisOptions } from '@cogitator-ai/swarms';
 export const DEFAULT_QUEUE_NAME = 'cogitator-jobs';
-/**
- * Connection settings from the queue's redis config: `url` first (`redis://` or `rediss://` for
- * TLS, `user:password@`, a `/db` path), then the explicit fields, which win.
- *
- * @throws Error for a url that is not redis:// or rediss://, or a db path that is not a number
- */
-export function resolveRedisOptions(redis) {
-    let fromUrl = {};
-    if (redis.url) {
-        const url = new URL(redis.url);
-        if (url.protocol !== 'redis:' && url.protocol !== 'rediss:') {
-            throw new Error(`Redis url must start with redis:// or rediss://, got ${url.protocol}//`);
-        }
-        const dbPath = url.pathname.replace(/^\//, '');
-        if (dbPath !== '' && !/^\d+$/.test(dbPath)) {
-            throw new Error(`Redis url database must be a number, got "${dbPath}"`);
-        }
-        fromUrl = {
-            ...(url.hostname && { host: url.hostname }),
-            ...(url.port && { port: Number(url.port) }),
-            ...(url.username && { username: decodeURIComponent(url.username) }),
-            ...(url.password && { password: decodeURIComponent(url.password) }),
-            ...(dbPath !== '' && { db: Number(dbPath) }),
-            tlsFromUrl: url.protocol === 'rediss:',
-        };
-    }
-    const tls = redis.tls ?? fromUrl.tlsFromUrl ?? false;
-    const username = redis.username ?? fromUrl.username;
-    const password = redis.password ?? fromUrl.password;
-    const db = redis.db ?? fromUrl.db;
-    return {
-        host: redis.host ?? fromUrl.host ?? 'localhost',
-        port: redis.port ?? fromUrl.port ?? 6379,
-        ...(username !== undefined && { username }),
-        ...(password !== undefined && { password }),
-        ...(db !== undefined && { db }),
-        ...(tls && { tls: {} }),
-    };
-}
 /**
  * BullMQ key prefix; cluster mode needs a hash tag so all queue keys share a slot.
  */
 export function queuePrefix(redis) {
     return redis.cluster ? '{cogitator}' : 'cogitator';
 }
-/**
- * What every cluster node connection needs from the config: the credentials and TLS of the url
- * and the explicit fields, without a host, port or database (a cluster has no database).
- */
-function clusterNodeOptions(redis) {
-    const { host: _host, port: _port, db: _db, ...rest } = resolveRedisOptions(redis);
-    return rest;
-}
 /**
  * Connection for BullMQ queues and workers. Blocking (worker) connections must not limit
  * retries per request, as BullMQ requires. Cluster instances are shared with BullMQ, which

package/dist/connection.js.map

@@ -1 +1 @@
-{"version":3,"file":"connection.js","sourceRoot":"","sources":["../src/connection.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,OAAO,EAAE,KAAK,EAAE,MAAM,SAAS,CAAC;AAIzC,MAAM,CAAC,MAAM,kBAAkB,GAAG,gBAAgB,CAAC;AAYnD;;;;;GAKG;AACH,MAAM,UAAU,mBAAmB,CAAC,KAA4B;IAC9D,IAAI,OAAO,GAA+D,EAAE,CAAC;IAC7E,IAAI,KAAK,CAAC,GAAG,EAAE,CAAC;QACd,MAAM,GAAG,GAAG,IAAI,GAAG,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC;QAC/B,IAAI,GAAG,CAAC,QAAQ,KAAK,QAAQ,IAAI,GAAG,CAAC,QAAQ,KAAK,SAAS,EAAE,CAAC;YAC5D,MAAM,IAAI,KAAK,CAAC,wDAAwD,GAAG,CAAC,QAAQ,IAAI,CAAC,CAAC;QAC5F,CAAC;QACD,MAAM,MAAM,GAAG,GAAG,CAAC,QAAQ,CAAC,OAAO,CAAC,KAAK,EAAE,EAAE,CAAC,CAAC;QAC/C,IAAI,MAAM,KAAK,EAAE,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,MAAM,CAAC,EAAE,CAAC;YAC3C,MAAM,IAAI,KAAK,CAAC,6CAA6C,MAAM,GAAG,CAAC,CAAC;QAC1E,CAAC;QACD,OAAO,GAAG;YACR,GAAG,CAAC,GAAG,CAAC,QAAQ,IAAI,EAAE,IAAI,EAAE,GAAG,CAAC,QAAQ,EAAE,CAAC;YAC3C,GAAG,CAAC,GAAG,CAAC,IAAI,IAAI,EAAE,IAAI,EAAE,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,EAAE,CAAC;YAC3C,GAAG,CAAC,GAAG,CAAC,QAAQ,IAAI,EAAE,QAAQ,EAAE,kBAAkB,CAAC,GAAG,CAAC,QAAQ,CAAC,EAAE,CAAC;YACnE,GAAG,CAAC,GAAG,CAAC,QAAQ,IAAI,EAAE,QAAQ,EAAE,kBAAkB,CAAC,GAAG,CAAC,QAAQ,CAAC,EAAE,CAAC;YACnE,GAAG,CAAC,MAAM,KAAK,EAAE,IAAI,EAAE,EAAE,EAAE,MAAM,CAAC,MAAM,CAAC,EAAE,CAAC;YAC5C,UAAU,EAAE,GAAG,CAAC,QAAQ,KAAK,SAAS;SACvC,CAAC;IACJ,CAAC;IACD,MAAM,GAAG,GAAG,KAAK,CAAC,GAAG,IAAI,OAAO,CAAC,UAAU,IAAI,KAAK,CAAC;IACrD,MAAM,QAAQ,GAAG,KAAK,CAAC,QAAQ,IAAI,OAAO,CAAC,QAAQ,CAAC;IACpD,MAAM,QAAQ,GAAG,KAAK,CAAC,QAAQ,IAAI,OAAO,CAAC,QAAQ,CAAC;IACpD,MAAM,EAAE,GAAG,KAAK,CAAC,EAAE,IAAI,OAAO,CAAC,EAAE,CAAC;IAClC,OAAO;QACL,IAAI,EAAE,KAAK,CAAC,IAAI,IAAI,OAAO,CAAC,IAAI,IAAI,WAAW;QAC/C,IAAI,EAAE,KAAK,CAAC,IAAI,IAAI,OAAO,CAAC,IAAI,IAAI,IAAI;QACxC,GAAG,CAAC,QAAQ,KAAK,SAAS,IAAI,EAAE,QAAQ,EAAE,CAAC;QAC3C,GAAG,CAAC,QAAQ,KAAK,SAAS,IAAI,EAAE,QAAQ,EAAE,CAAC;QAC3C,GAAG,CAAC,EAAE,KAAK,SAAS,IAAI,EAAE,EAAE,EAAE,CAAC;QAC/B,GAAG,CAAC,GAAG,IAAI,EAAE,GAAG,EAAE,EAAE,EAAE,CAAC;KACxB,CAAC;AACJ,CAAC;AAMD;;GAEG;AACH,MAAM,UAAU,WAAW,CAAC,KAA4B;IACtD,OAAO,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,aAAa,CAAC,CAAC,CAAC,WAAW,CAAC;AACrD,CAAC;AAQD;;;GAGG;AACH,SAAS,kBAAkB,CACzB,KAA4B;IAE5B,MAAM,EAAE,IAAI,EAAE,KAAK,EAAE,IAAI,EAAE,KAAK,EAAE,EAAE,EAAE,GAAG,EAAE,GAAG,IAAI,EAAE,GAAG,mBAAmB,CAAC,KAAK,CAAC,CAAC;IAClF,OAAO,IAAI,CAAC;AACd,CAAC;AAED;;;;GAIG;AACH,MAAM,UAAU,oBAAoB,CAClC,KAA4B,EAC5B,UAAkC,EAAE;IAEpC,MAAM,oBAAoB,GAAG,OAAO,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,SAAS,CAAC;IAEjE,IAAI,KAAK,CAAC,OAAO,EAAE,CAAC;QAClB,IAAI,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;YACrC,MAAM,IAAI,KAAK,CAAC,wDAAwD,CAAC,CAAC;QAC5E,CAAC;QACD,MAAM,OAAO,GAAG,IAAI,OAAO,CAAC,KAAK,CAAC,OAAO,CAAC,KAAK,EAAE;YAC/C,WAAW,EAAE,IAAI;YACjB,YAAY,EAAE,EAAE,GAAG,kBAAkB,CAAC,KAAK,CAAC,EAAE,oBAAoB,EAAE;SACrE,CAAC,CAAC;QACH,OAAO;YACL,UAAU,EAAE,OAAO;YACnB,OAAO,EAAE,KAAK,IAAI,EAAE;gBAClB,IAAI,OAAO,CAAC,MAAM,KAAK,KAAK;oBAAE,OAAO;gBACrC,IAAI,OAAO,CAAC,MAAM,KAAK,MAAM,EAAE,CAAC;oBAC9B,OAAO,CAAC,UAAU,EAAE,CAAC;oBACrB,OAAO;gBACT,CAAC;gBACD,MAAM,OAAO,CAAC,IAAI,EAAE,CAAC;YACvB,CAAC;SACF,CAAC;IACJ,CAAC;IAED,OAAO;QACL,UAAU,EAAE,EAAE,GAAG,mBAAmB,CAAC,KAAK,CAAC,EAAE,oBAAoB,EAAE;QACnE,OAAO,EAAE,KAAK,IAAI,EAAE,GAAE,CAAC;KACxB,CAAC;AACJ,CAAC;AAED;;GAEG;AACH,MAAM,UAAU,iBAAiB,CAAC,KAA4B;IAC5D,IAAI,KAAK,CAAC,OAAO,EAAE,CAAC;QAClB,IAAI,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;YACrC,MAAM,IAAI,KAAK,CAAC,wDAAwD,CAAC,CAAC;QAC5E,CAAC;QACD,OAAO,IAAI,OAAO,CAAC,KAAK,CAAC,OAAO,CAAC,KAAK,EAAE,EAAE,YAAY,EAAE,kBAAkB,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;IACvF,CAAC;IAED,OAAO,IAAI,KAAK,CAAC,mBAAmB,CAAC,KAAK,CAAC,CAAC,CAAC;AAC/C,CAAC"}
+{"version":3,"file":"connection.js","sourceRoot":"","sources":["../src/connection.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,OAAO,EAAE,KAAK,EAAE,MAAM,SAAS,CAAC;AAEzC,OAAO,EAAE,kBAAkB,EAAE,mBAAmB,EAAE,MAAM,sBAAsB,CAAC;AAG/E,OAAO,EAAE,mBAAmB,EAA+B,MAAM,sBAAsB,CAAC;AAExF,MAAM,CAAC,MAAM,kBAAkB,GAAG,gBAAgB,CAAC;AAMnD;;GAEG;AACH,MAAM,UAAU,WAAW,CAAC,KAA4B;IACtD,OAAO,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,aAAa,CAAC,CAAC,CAAC,WAAW,CAAC;AACrD,CAAC;AAQD;;;;GAIG;AACH,MAAM,UAAU,oBAAoB,CAClC,KAA4B,EAC5B,UAAkC,EAAE;IAEpC,MAAM,oBAAoB,GAAG,OAAO,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,SAAS,CAAC;IAEjE,IAAI,KAAK,CAAC,OAAO,EAAE,CAAC;QAClB,IAAI,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;YACrC,MAAM,IAAI,KAAK,CAAC,wDAAwD,CAAC,CAAC;QAC5E,CAAC;QACD,MAAM,OAAO,GAAG,IAAI,OAAO,CAAC,KAAK,CAAC,OAAO,CAAC,KAAK,EAAE;YAC/C,WAAW,EAAE,IAAI;YACjB,YAAY,EAAE,EAAE,GAAG,kBAAkB,CAAC,KAAK,CAAC,EAAE,oBAAoB,EAAE;SACrE,CAAC,CAAC;QACH,OAAO;YACL,UAAU,EAAE,OAAO;YACnB,OAAO,EAAE,KAAK,IAAI,EAAE;gBAClB,IAAI,OAAO,CAAC,MAAM,KAAK,KAAK;oBAAE,OAAO;gBACrC,IAAI,OAAO,CAAC,MAAM,KAAK,MAAM,EAAE,CAAC;oBAC9B,OAAO,CAAC,UAAU,EAAE,CAAC;oBACrB,OAAO;gBACT,CAAC;gBACD,MAAM,OAAO,CAAC,IAAI,EAAE,CAAC;YACvB,CAAC;SACF,CAAC;IACJ,CAAC;IAED,OAAO;QACL,UAAU,EAAE,EAAE,GAAG,mBAAmB,CAAC,KAAK,CAAC,EAAE,oBAAoB,EAAE;QACnE,OAAO,EAAE,KAAK,IAAI,EAAE,GAAE,CAAC;KACxB,CAAC;AACJ,CAAC;AAED;;GAEG;AACH,MAAM,UAAU,iBAAiB,CAAC,KAA4B;IAC5D,IAAI,KAAK,CAAC,OAAO,EAAE,CAAC;QAClB,IAAI,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;YACrC,MAAM,IAAI,KAAK,CAAC,wDAAwD,CAAC,CAAC;QAC5E,CAAC;QACD,OAAO,IAAI,OAAO,CAAC,KAAK,CAAC,OAAO,CAAC,KAAK,EAAE,EAAE,YAAY,EAAE,kBAAkB,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;IACvF,CAAC;IAED,OAAO,IAAI,KAAK,CAAC,mBAAmB,CAAC,KAAK,CAAC,CAAC,CAAC;AAC/C,CAAC"}

package/dist/distributed-swarm-worker.d.ts

@@ -2,18 +2,18 @@
  * Worker node for distributed swarms.
  *
  * Consumes agent turns dispatched by `DistributedSwarmCoordinator` (from
- * `@cogitator-ai/swarms`), runs them with the worker's Cogitator and tools, and publishes
- * each result back to the coordinator's results channel.
+ * `@cogitator-ai/swarms`), runs them with the worker's Cogitator and tools, and hands each
+ * result back to the coordinator. A turn moves from the queue to this worker's own list while
+ * it runs, so a turn of a worker that died goes back to the queue for another one.
  */
+import type { RedisConnectionConfig } from '@cogitator-ai/types';
 import type { SwarmAgentJobPayload, SwarmAgentJobResult, WorkerRuntime } from './types.js';
 export interface DistributedSwarmWorkerConfig extends WorkerRuntime {
-    /** Redis holding the swarm job queue (must match the swarm's `distributed.redis`) */
-    redis?: {
-        host?: string;
-        port?: number;
-        password?: string;
-        db?: number;
-    };
+    /**
+     * Redis holding the swarm job queue (must match the swarm's `distributed.redis`): a url, host
+     * and port, TLS, or a cluster. Default: localhost:6379
+     */
+    redis?: RedisConnectionConfig;
     /** Swarm key prefix (must match `distributed.redis.keyPrefix`, default: 'swarm') */
     keyPrefix?: string;
     /** Job queue name (must match `distributed.queue`, default: 'swarm-agent-jobs') */
@@ -28,39 +28,98 @@ export interface DistributedSwarmWorkerConfig extends WorkerRuntime {
      * cancelled turn is aborted and its result is not published
      */
     cancelCheckInterval?: number;
+    /**
+     * How often, in ms, the worker tells the queue it is alive and returns the turns of workers
+     * that stopped doing so to the queue (default: 5000). A worker silent for three intervals
+     * counts as gone, so a turn that runs past that on a stalled worker can run twice
+     */
+    heartbeatInterval?: number;
 }
 export interface DistributedSwarmWorkerEvents {
     onJobStarted?: (job: SwarmAgentJobPayload) => void;
     onJobCompleted?: (job: SwarmAgentJobPayload, result: SwarmAgentJobResult) => void;
     onJobFailed?: (job: SwarmAgentJobPayload, error: Error) => void;
-    /** The coordinator gave up on the turn: it was skipped, or aborted while it ran */
+    /**
+     * The coordinator gave up on the turn or already has its result: it was skipped, or aborted
+     * while it ran
+     */
     onJobCancelled?: (job: SwarmAgentJobPayload) => void;
+    /**
+     * Turns went back to the queue: those of a worker that stopped beating, or ones on this
+     * worker's own list it never got (`workerId` is then its own)
+     */
+    onJobsRecovered?: (workerId: string, count: number) => void;
     onError?: (error: Error) => void;
 }
 export declare class DistributedSwarmWorker {
     private readonly config;
     private readonly events;
     private readonly runtime;
-    private readonly queueKey;
+    private readonly keys;
+    /** This worker's id in the queue's worker set, new for every start */
+    private workerId;
     private consumer?;
     private publisher?;
     private loop?;
+    /** The next beat, scheduled once the one before it finished */
+    private beat?;
+    /** The beat running now, which `stop()` waits for */
+    private beating?;
+    /** Whether the worker beats: from `start()` until `stop()` saw its last turn finish */
+    private alive;
     private running;
     private readonly active;
+    /** The turns this worker took and runs, as moved: one entry per copy */
+    private readonly holding;
+    /** Turns on this worker's list it did not run at the last beat, with how many copies */
+    private strays;
     constructor(config?: DistributedSwarmWorkerConfig, events?: DistributedSwarmWorkerEvents);
     /**
-     * Start consuming jobs. Resolves once the worker is connected.
+     * Start consuming jobs. Resolves once the worker is connected, has joined the queue's workers
+     * and has returned the turns of workers that are gone to the queue.
      */
     start(): Promise<void>;
     /**
-     * Stop taking new jobs, wait for in-flight jobs to finish and close connections.
+     * Stop taking new jobs, wait for in-flight jobs to finish, hand back to the queue whatever is
+     * still on this worker's list, leave the queue's workers and close connections.
      */
     stop(): Promise<void>;
+    /** This worker's id in the queue's worker set while it runs */
+    getWorkerId(): string;
     isRunning(): boolean;
     getActiveJobCount(): number;
     private consume;
+    /**
+     * Beat once `heartbeatInterval` ms from now, and again that long after it finished: one beat at
+     * a time, so a slow Redis never runs two `reclaim()` at once or shortens the grace between two
+     */
+    private scheduleBeat;
+    private heartbeatInterval;
+    /** Tell the queue this worker is alive: its heartbeat first, so no one takes it for gone */
+    private heartbeat;
+    /**
+     * Hand back to the queue the turns on this worker's own list it does not run, at the second
+     * beat in a row that finds them there: a BLMOVE whose reply a dropped connection lost (and
+     * ioredis sent again) moved a turn this worker never got. One beat of grace lets a reply that
+     * is still on its way arrive first
+     */
+    private reclaim;
+    /** Return the turns of every worker whose heartbeat is gone to the queue */
+    private recover;
+    /**
+     * Move the turns on a worker's list back to the head of the queue, one at a time so two
+     * workers recovering it at once never move a turn twice
+     *
+     * @returns how many turns this call moved
+     */
+    private requeue;
+    private reportError;
     private handle;
-    /** Whether the coordinator gave up on the job (only coordinators that name a cancel set). */
+    /**
+     * Whether the coordinator gave up on the job (only coordinators that name a cancel set), or a
+     * worker answered it already, such as one that died after answering and before taking the job
+     * off its list
+     */
     private isCancelled;
     /** Abort `controller` once the coordinator cancels the job while it runs. */
     private watchCancellation;

package/dist/distributed-swarm-worker.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"distributed-swarm-worker.d.ts","sourceRoot":"","sources":["../src/distributed-swarm-worker.ts"],"names":[],"mappings":"AAAA;;;;;;GAMG;AAIH,OAAO,KAAK,EAAE,oBAAoB,EAAE,mBAAmB,EAAE,aAAa,EAAE,MAAM,SAAS,CAAC;AAKxF,MAAM,WAAW,4BAA6B,SAAQ,aAAa;IACjE,qFAAqF;IACrF,KAAK,CAAC,EAAE;QACN,IAAI,CAAC,EAAE,MAAM,CAAC;QACd,IAAI,CAAC,EAAE,MAAM,CAAC;QACd,QAAQ,CAAC,EAAE,MAAM,CAAC;QAClB,EAAE,CAAC,EAAE,MAAM,CAAC;KACb,CAAC;IACF,oFAAoF;IACpF,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,mFAAmF;IACnF,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,qEAAqE;IACrE,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,uFAAuF;IACvF,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB;;;;OAIG;IACH,mBAAmB,CAAC,EAAE,MAAM,CAAC;CAC9B;AAED,MAAM,WAAW,4BAA4B;IAC3C,YAAY,CAAC,EAAE,CAAC,GAAG,EAAE,oBAAoB,KAAK,IAAI,CAAC;IACnD,cAAc,CAAC,EAAE,CAAC,GAAG,EAAE,oBAAoB,EAAE,MAAM,EAAE,mBAAmB,KAAK,IAAI,CAAC;IAClF,WAAW,CAAC,EAAE,CAAC,GAAG,EAAE,oBAAoB,EAAE,KAAK,EAAE,KAAK,KAAK,IAAI,CAAC;IAChE,mFAAmF;IACnF,cAAc,CAAC,EAAE,CAAC,GAAG,EAAE,oBAAoB,KAAK,IAAI,CAAC;IACrD,OAAO,CAAC,EAAE,CAAC,KAAK,EAAE,KAAK,KAAK,IAAI,CAAC;CAClC;AAgBD,qBAAa,sBAAsB;IACjC,OAAO,CAAC,QAAQ,CAAC,MAAM,CAA+B;IACtD,OAAO,CAAC,QAAQ,CAAC,MAAM,CAA+B;IACtD,OAAO,CAAC,QAAQ,CAAC,OAAO,CAA0B;IAClD,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAS;IAClC,OAAO,CAAC,QAAQ,CAAC,CAAQ;IACzB,OAAO,CAAC,SAAS,CAAC,CAAQ;IAC1B,OAAO,CAAC,IAAI,CAAC,CAAgB;IAC7B,OAAO,CAAC,OAAO,CAAS;IACxB,OAAO,CAAC,QAAQ,CAAC,MAAM,CAA4B;gBAGjD,MAAM,GAAE,4BAAiC,EACzC,MAAM,GAAE,4BAAiC;IAW3C;;OAEG;IACG,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC;IA2B5B;;OAEG;IACG,IAAI,IAAI,OAAO,CAAC,IAAI,CAAC;IAc3B,SAAS,IAAI,OAAO;IAIpB,iBAAiB,IAAI,MAAM;YAIb,OAAO;YA6BP,MAAM;IAiDpB,6FAA6F;YAC/E,WAAW;IAWzB,6EAA6E;IAC7E,OAAO,CAAC,iBAAiB;CAqB1B"}
+{"version":3,"file":"distributed-swarm-worker.d.ts","sourceRoot":"","sources":["../src/distributed-swarm-worker.ts"],"names":[],"mappings":"AAAA;;;;;;;GAOG;AAWH,OAAO,KAAK,EAAE,qBAAqB,EAAE,MAAM,qBAAqB,CAAC;AACjE,OAAO,KAAK,EAAE,oBAAoB,EAAE,mBAAmB,EAAE,aAAa,EAAE,MAAM,SAAS,CAAC;AAKxF,MAAM,WAAW,4BAA6B,SAAQ,aAAa;IACjE;;;OAGG;IACH,KAAK,CAAC,EAAE,qBAAqB,CAAC;IAC9B,oFAAoF;IACpF,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,mFAAmF;IACnF,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,qEAAqE;IACrE,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,uFAAuF;IACvF,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB;;;;OAIG;IACH,mBAAmB,CAAC,EAAE,MAAM,CAAC;IAC7B;;;;OAIG;IACH,iBAAiB,CAAC,EAAE,MAAM,CAAC;CAC5B;AAED,MAAM,WAAW,4BAA4B;IAC3C,YAAY,CAAC,EAAE,CAAC,GAAG,EAAE,oBAAoB,KAAK,IAAI,CAAC;IACnD,cAAc,CAAC,EAAE,CAAC,GAAG,EAAE,oBAAoB,EAAE,MAAM,EAAE,mBAAmB,KAAK,IAAI,CAAC;IAClF,WAAW,CAAC,EAAE,CAAC,GAAG,EAAE,oBAAoB,EAAE,KAAK,EAAE,KAAK,KAAK,IAAI,CAAC;IAChE;;;OAGG;IACH,cAAc,CAAC,EAAE,CAAC,GAAG,EAAE,oBAAoB,KAAK,IAAI,CAAC;IACrD;;;OAGG;IACH,eAAe,CAAC,EAAE,CAAC,QAAQ,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,KAAK,IAAI,CAAC;IAC5D,OAAO,CAAC,EAAE,CAAC,KAAK,EAAE,KAAK,KAAK,IAAI,CAAC;CAClC;AAgCD,qBAAa,sBAAsB;IACjC,OAAO,CAAC,QAAQ,CAAC,MAAM,CAA+B;IACtD,OAAO,CAAC,QAAQ,CAAC,MAAM,CAA+B;IACtD,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAuE;IAC/F,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAiB;IACtC,sEAAsE;IACtE,OAAO,CAAC,QAAQ,CAAM;IACtB,OAAO,CAAC,QAAQ,CAAC,CAAa;IAC9B,OAAO,CAAC,SAAS,CAAC,CAAa;IAC/B,OAAO,CAAC,IAAI,CAAC,CAAgB;IAC7B,+DAA+D;IAC/D,OAAO,CAAC,IAAI,CAAC,CAAgC;IAC7C,qDAAqD;IACrD,OAAO,CAAC,OAAO,CAAC,CAAgB;IAChC,uFAAuF;IACvF,OAAO,CAAC,KAAK,CAAS;IACtB,OAAO,CAAC,OAAO,CAAS;IACxB,OAAO,CAAC,QAAQ,CAAC,MAAM,CAA4B;IACnD,wEAAwE;IACxE,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAgB;IACxC,wFAAwF;IACxF,OAAO,CAAC,MAAM,CAA6B;gBAGzC,MAAM,GAAE,4BAAiC,EACzC,MAAM,GAAE,4BAAiC;IAW3C;;;OAGG;IACG,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC;IA0B5B;;;OAGG;IACG,IAAI,IAAI,OAAO,CAAC,IAAI,CAAC;IA2B3B,+DAA+D;IAC/D,WAAW,IAAI,MAAM;IAIrB,SAAS,IAAI,OAAO;IAIpB,iBAAiB,IAAI,MAAM;YAIb,OAAO;IA2CrB;;;OAGG;IACH,OAAO,CAAC,YAAY;IAapB,OAAO,CAAC,iBAAiB;IAIzB,4FAA4F;YAC9E,SAAS;IAWvB;;;;;OAKG;YACW,OAAO;IA6BrB,4EAA4E;YAC9D,OAAO;IAiBrB;;;;;OAKG;YACW,OAAO;IAYrB,OAAO,CAAC,WAAW;YAIL,MAAM;IAiDpB;;;;OAIG;YACW,WAAW;IAYzB,6EAA6E;IAC7E,OAAO,CAAC,iBAAiB;CAqB1B"}