@pauldeng/node-red-contrib-bullmq 1.0.2 → 2.0.0

package/docs/REFERENCE_MAP.md

@@ -3,11 +3,11 @@
 ## Runtime
 
 - `bull-queue.js`: Node-RED registration and runtime glue.
-- `lib/connections.js`: Redis deployment normalization and ioredis descriptors.
-- `lib/scheduler.js`: legacy repeat-cron to BullMQ Job Scheduler normalization.
-- `lib/commands.js`: `bull cmd` dispatch.
+- `lib/connections.js`: backend selection, Redis deployment normalization and ioredis descriptors, and PostgreSQL pool configuration.
+- `lib/scheduler.js`: BullMQ v6 Job Scheduler id handling and serialization.
+- `lib/commands.js`: `bullmq cmd` dispatch.
 - `lib/serialization.js`: message-safe BullMQ serialization.
-- `lib/acknowledgements.js`: manual-acknowledgement registry for `bull run`/`bull job`. Entries self-remove on settle.
+- `lib/acknowledgements.js`: manual-acknowledgement registry for `bullmq run`/`bullmq job`. Entries self-remove on settle.
 
 ## Editor
 
@@ -17,16 +17,26 @@
 ## Tests
 
 - `test/package-contract.test.js`: dependency and runtime import contract.
-- `test/connections.test.js`: Redis topology option normalization.
-- `test/scheduler.test.js`: repeat scheduler compatibility.
+- `test/connections.test.js`: backend selection and Redis/PostgreSQL config normalization.
+- `test/scheduler.test.js`: native Job Scheduler id and serialization contracts.
 - `test/commands.test.js`: command dispatch behavior.
 - `test/acknowledgements.test.js`: manual-acknowledgement registry lifecycle and leak prevention.
+- `test/shutdown.test.js`: BullMQ/ioredis resource ownership, partial redeploy cleanup, and concurrent shutdown.
+- `test/async-style.test.js`: bounded-close async implementation constraints.
 - `test/node-red-registration.test.js`: Node-RED node type registration.
 - `test/editor-contract.test.js`: static editor surface.
 - `test/docs-contract.test.js`: required docs and examples.
+- `test/serialization.test.js`: message-safe job serialization, including that no lock token escapes.
+- `test/telemetry.test.js`: opt-in telemetry wiring, including the QueueEvents exclusion.
 - `test/docker-matrix-contract.test.js`: Docker deployment fixture and runner contract.
-- `test/integration-standalone.test.js`: opt-in local Redis Node-RED runtime flow tests for add/run, manual acknowledgement, repeat schedulers, delayed jobs, priorities, rate limits, deduplication events, and flow producer jobs.
-- `test/integration-deployment.test.js`: opt-in external Redis deployment flow test used by Docker and MemoryDB.
+- `test/redis-characterization.test.js`: pins the Redis descriptors and BullMQ options against backend-seam drift.
+- `test/postgres-wiring.test.js`: PostgreSQL pool config, backend factory positions, and error categories.
+- `test/integration-postgres.test.js`: opt-in PostgreSQL migrations, custom schema, and close behavior against a container.
+- `test/integration-postgres-tls.test.js`: opt-in PostgreSQL TLS connection and `ssl` object shape.
+- `test/integration-standalone.test.js`: opt-in local Redis/PostgreSQL Node-RED runtime flows, standing command/action coverage, schedulers, delayed jobs, events, and flow producer jobs.
+- `test/integration-deployment.test.js`: opt-in external Redis/PostgreSQL deployment flow test used by Docker and MemoryDB.
+- `test/helpers/stores.js`: local Redis/PostgreSQL integration fixtures and backend adapters.
+- `test/playwright/editor.spec.js`: editor dialog behavior in a real Node-RED browser session.
 
 ## Deployment Test Fixtures
 
@@ -38,14 +48,23 @@
 - `test/deployments/cluster-tls/`: Redis Cluster with TLS.
 - `test/deployments/sentinel-auth/`: Redis Sentinel with data-node ACL auth.
 - `test/deployments/sentinel-tls/`: Redis Sentinel with TLS for data-node and Sentinel connections.
+- `test/deployments/postgres-plain/`: PostgreSQL without TLS.
+- `test/deployments/postgres-tls/`: PostgreSQL with TLS.
 - `test/deployments/tls-certs/`: local self-signed certificates for Docker TLS fixtures.
 
+## Agent Docs
+
+- `AGENTS.md`: entry point; `CLAUDE.md` imports it.
+- `docs/RULES.md`: hard constraints.
+- `docs/CHANGE_WORKFLOW.md`: change procedure.
+
 ## User Docs
 
 - `README.md`: overview and installation.
 - `docs/NODE_GUIDE.md`: node behavior.
 - `docs/COMMANDS.md`: command reference.
-- `docs/CONNECTIONS.md`: Redis deployments.
-- `docs/MIGRATION.md`: Bull v4 migration.
+- `docs/CONNECTIONS.md`: Redis and PostgreSQL backends.
+- `docs/TELEMETRY.md`: opt-in OpenTelemetry tracing and metrics.
+- `docs/MIGRATION.md`: breaking upgrade from older package versions.
 - `docs/TESTING.md`: verification plan.
 - `docs/TROUBLESHOOTING.md`: operational issues.

package/docs/RELEASE.md

@@ -13,9 +13,11 @@ used here only for the one-time first publish.)
 
 1. Confirm `package.json` has the intended `version` and the name `@pauldeng/node-red-contrib-bullmq`.
 2. Confirm the GitHub repository is `https://github.com/pauldeng/node-red-contrib-bullmq`.
-3. Confirm BullMQ is pinned to exactly `5.80.2`.
-4. Confirm no examples, docs, fixtures, or logs contain Redis, Sentinel, or MemoryDB secrets.
-5. Update `CHANGELOG.md` for the new version.
+3. Confirm BullMQ is pinned to exactly `6.3.1`.
+4. Confirm no examples, docs, fixtures, or logs contain Redis, Sentinel, MemoryDB, or PostgreSQL secrets. A shipped example flow carries no password, CA, or client key at all.
+5. Confirm `pg` is declared only as an optional peer dependency, never a dependency: it must stay uninstalled for Redis-only users.
+6. Update `CHANGELOG.md` for the new version.
+7. For a major version bump, confirm `CHANGELOG.md` names every breaking change and `docs/MIGRATION.md` covers the upgrade path before tagging.
 
 ## 2. Local verification
 
@@ -29,12 +31,23 @@ npm audit --omit=dev --audit-level=moderate
 npm pack --dry-run                        # confirm the tarball is clean
 ```
 
-For deployment changes, also run:
+For deployment changes, also run the Docker matrix, which covers the Redis
+topologies plus the `postgres-plain` and `postgres-tls` fixtures:
 
 ```sh
 npm run test:deployments
 ```
 
+For backend changes, also run the live backend suites. `npm run test:integration`
+covers both backends when Docker is available and reports a reason when it is
+not; the PostgreSQL schema and TLS suites are separate:
+
+```sh
+npm run test:integration
+BULLMQ_INTEGRATION_POSTGRES=1 node --test test/integration-postgres.test.js
+BULLMQ_INTEGRATION_POSTGRES=1 node --test test/integration-postgres-tls.test.js
+```
+
 MemoryDB verification is opt-in and environment-only:
 
 ```sh
@@ -56,7 +69,7 @@ Do these once, before the first release.
 - [ ] **Branch ruleset** on `master` (Settings -> Rules -> Rulesets): require a pull request with at least one review, require the CI status checks to pass, block force pushes, restrict deletions. Optionally require linear history and signed commits.
 - [ ] **Code security and analysis** (Settings): enable Dependabot alerts, Dependabot security updates, secret scanning, push protection, and private vulnerability reporting.
 - [ ] **Workflow permissions** (Settings -> Actions -> General): set the default `GITHUB_TOKEN` to read-only.
-- [ ] **CodeQL**: either rely on `.github/workflows/codeql.yml`, or enable Code scanning "default setup" in the Security tab.
+- [ ] **CodeQL**: enable Code scanning "default setup" in the Security tab.
 - [ ] **`release` environment** (Settings -> Environments -> New environment -> `release`): add yourself as a required reviewer so `publish.yml` waits for manual approval.
 
 ## 4. First publish (one time, manual)

package/docs/RULES.md

@@ -0,0 +1,33 @@
+# Rules
+
+Hard constraints. Each one exists because breaking it breaks a released contract, loses jobs, or leaks a secret. Everything not listed here is a normal judgment call.
+
+## Public Surface
+
+- Register only the BullMQ v6 node types: `bullmq-queue-server`, `bullmq cmd`, `bullmq run`, `bullmq job`, `bullmq events`, and `bullmq flow`.
+- Accept only `msg.cmd` for command dispatch and only the native Job Scheduler command names and fields.
+- `msg.cmd` dispatch maps to explicit BullMQ calls only. Never expose arbitrary method names through it.
+- Record unsupported BullMQ behavior with a reason instead of silently omitting it.
+
+## Dependencies
+
+- BullMQ is pinned to an exact version. Do not widen the range; `package.json` holds the value and `test/package-contract.test.js` enforces it.
+- Changing the BullMQ pin means re-verifying the connection-readiness and shutdown paths against BullMQ's own source, not just running the suite. Three separate defects in the v5 to v6 upgrade came from behavior invisible at the public API: `close()` and `disconnect()` both await a connection-ready promise that never settles while Redis is unreachable, `Worker.close()` clears its lock-renewal and stalled-check timers only after the step that hangs, and `increaseMaxListeners` computes `getMaxListeners() + n`, which turns an unlimited emitter into a cap of 3. The code that works around these is in `bull-queue.js` (`forceDisconnect`, `closeBudgetFor` — including the PostgreSQL budget beyond pg's connection timeout — and the config node's single backend watcher, which replaced a per-node listener budget) and `lib/connections.js` (`skipWaitingForReady`, Redis-only: `createPostgresBackend` never reads it); each carries a comment naming the measured behavior it depends on. Read those, then confirm they still hold.
+- The shutdown escape hatch is tied to the **Redis adapter**, not to the pin alone: `forceDisconnect` reaches into `backend.connection._client` because `RedisQueueBackend` is the only installed backend exposing a raw handle. The PostgreSQL adapter needs a different answer and has one — no raw disconnect, and a close budget that outlasts pg's own connection timeout instead of capping below it. Hold both to the same standard: measured against a real unreachable server, commented with what was measured, and re-verified when the pin moves. What the suite proves automatically is the shape, not the wall-clock number: `test/shutdown.test.js` drives ten redeploys against a real local blackhole socket and asserts that the producer command rejects, that close outlives the Redis-only fast budget, and that no client socket survives (`process.getActiveResourcesInfo()`). It shortens `connectionTimeoutMillis` to do so, because ten full-length closes would be a hundred-second test. The full-length number is therefore a **manual** measurement, not a regression test: with the shipped 10-second `connectionTimeoutMillis`, `Queue`, `Worker`, `QueueEvents`, and `FlowProducer` each settled on their own in about 9.8 seconds against a blackholed server, leaving no active handles. Re-measure it by hand when the pin moves; a green suite does not re-confirm it.
+- Do not reintroduce `bull` or `sprintf-js`.
+- The runtime floor is Node.js 22.9 with Node-RED 5.x, for the published package and for local development alike.
+
+## Secrets
+
+- Credentials belong in Node-RED credentials or environment variables. Never in examples, docs, logs, fixtures, snapshots, or committed flows.
+- Never commit real credentials, Redis or PostgreSQL passwords, MemoryDB endpoints, or production private keys. One sanctioned exception: the local self-signed certificates in `test/deployments/tls-certs/` are committed Docker-only fixtures that grant access to nothing.
+- Editor credential fields must not export secrets into flow JSON.
+- MemoryDB tests read credentials only from environment variables.
+
+## Runtime Invariants
+
+- Never put BullMQ lock tokens in a Node-RED message.
+- Repeat scheduler lookup and removal use exact scheduler ids. Never substring-match scheduler keys, and never call the deprecated repeatable-job APIs.
+- Cluster and MemoryDB deployments default their BullMQ prefix to `{bull}`. A custom prefix must contain a Redis hash tag; `lib/connections.js` rejects one that does not. Redis-only: `prefix` is a Redis key-namespacing concept and is never sent on the PostgreSQL path.
+- Do not pass arbitrary ioredis options through messages or editor fields.
+- Keep TLS certificate verification on by default. Disable it only for a controlled deployment that cannot be given a trusted CA.

package/docs/TELEMETRY.md

@@ -0,0 +1,55 @@
+# Telemetry Guide
+
+## What It Gives You
+
+OpenTelemetry traces for producer, consumer, and internal BullMQ operations (`add`, `process`, `addFlow`, and similar) on Queue, Worker, and FlowProducer. Span names follow `{operation} {destination}`, where destination is normally the queue name (for example `process myQueue`, `addFlow myQueue`) but includes the job name for `add` (`add myQueue.myJobName`). Metrics are optional and separately switched.
+
+Telemetry is off by default. With the master switch off, the word `telemetry` never reaches a BullMQ constructor and `bullmq-otel` is never required.
+
+## Config Fields
+
+`bullmq-queue-server` has three telemetry fields:
+
+- `telemetry` (boolean, default `false`): master switch.
+- `telemetryServiceName` (string, default `""`): used as both the tracer name and the meter name; falls back to the queue name when blank.
+- `telemetryMetrics` (boolean, default `false`): sets `enableMetrics` on the telemetry client.
+
+One telemetry client is built per config node; the Queue, Worker, and FlowProducer backed by that config node share it.
+
+## Installation
+
+`bullmq-otel` is an optional peer dependency, pinned to `>=2.0.0`. It is not installed automatically. Install it alongside Node-RED:
+
+```sh
+npm install bullmq-otel
+```
+
+If `telemetry` is enabled and `bullmq-otel` cannot be required, the node reports one `node.error` naming `npm install bullmq-otel` and keeps running untraced; jobs keep flowing either way.
+
+## Host Requirements
+
+This package does not own the OpenTelemetry SDK or an exporter. The host process registers the `TracerProvider` (and, for metrics, the `MeterProvider`) — for example via `NODE_OPTIONS` auto-instrumentation, or SDK setup in Node-RED's `settings.js`.
+
+With nothing registered, the OpenTelemetry API no-ops. An enabled toggle without a configured host is harmless but exports nothing.
+
+## Metrics
+
+`telemetryMetrics` requires a `MeterProvider` registered _before_ the queue is first used — BullMQ reads it at telemetry-client construction time.
+
+BullMQ emits:
+
+- Counters, one per job state transition: `bullmq.jobs.completed`, `bullmq.jobs.failed`, `bullmq.jobs.delayed`, `bullmq.jobs.retried`, `bullmq.jobs.waiting`, `bullmq.jobs.waiting_children`. Attributes: `bullmq.queue.name`, `bullmq.job.name`, `bullmq.job.state`.
+- Histogram `bullmq.job.duration` (milliseconds), recorded alongside the counter above whenever the job has a `processedOn` timestamp. Same attributes.
+- Gauge `bullmq.queue.jobs`, the job count per state. Attributes: `bullmq.queue.name`, `bullmq.queue.jobs.state`. BullMQ only records this gauge when something calls `queue.recordJobCountsMetric()`; this package does not call it on a timer, so the gauge stays empty unless a host script invokes that method directly.
+
+## `bullmq events` Is Not Traced
+
+BullMQ 6.3.1 types `QueueEventsOptions` as `Omit<QueueBaseOptions, 'telemetry'>`. `QueueEvents` accepts no telemetry client at all, so `bullmq events` never emits spans or metrics regardless of the config-node toggles.
+
+## Zero-Dependency Alternative
+
+`msg.cmd` `exportPrometheusMetrics` (see `docs/COMMANDS.md`) needs no OpenTelemetry stack. Install nothing extra if Prometheus scraping is all you want.
+
+## Security
+
+Never put a secret in `telemetryServiceName` — it appears as an attribute on every span and every metric this package emits.

package/docs/TESTING.md

@@ -4,37 +4,48 @@
 
 ```sh
 npm test
+npm run format:check
 ```
 
-This runs built-in `node:test` suites for package metadata, connection normalization, scheduler compatibility, command dispatch, editor surface, registration, Docker fixture contracts, docs, and examples.
+`npm test` runs built-in `node:test` suites for package metadata, connection normalization, Job Scheduler commands, command dispatch, editor surface, registration, resource shutdown, async style, telemetry wiring, Docker fixture contracts, docs, and examples. The lifecycle checks live in `test/shutdown.test.js`; bounded-close implementation constraints live in `test/async-style.test.js`. `npm run format:check` runs Prettier. CI gates both on every pull request.
 
 ## Node-RED Runtime Tests
 
 Use `node-red-node-test-helper` for flow-level tests. Runtime coverage should load actual Node-RED flows for:
 
-- `bull cmd` success and failure paths;
-- `bull run` immediate and manual modes;
-- `bull job` acknowledgement actions;
-- `bull events`;
-- `bull flow`;
-- legacy flow compatibility.
+- `bullmq cmd` success and failure paths;
+- `bullmq run` immediate and manual modes;
+- `bullmq job` acknowledgement actions;
+- `bullmq events`;
+- `bullmq flow`;
+- BullMQ v6 Job Scheduler commands.
 
-The current standalone Redis integration suite is opt-in:
+The local Redis/PostgreSQL integration suite is opt-in:
 
 ```sh
 npm run test:integration
 ```
 
-It starts a temporary local `redis-server`, loads real Node-RED flows, and verifies:
+It runs each backend-neutral flow against a temporary local `redis-server`
+(falling back to a temporary Redis container) and a temporary PostgreSQL
+container. When neither backend prerequisite is available, its tests are
+reported as skipped with the reason rather than silently omitted.
+It verifies:
 
-- `bull cmd` add/run behavior;
-- manual `bull run` acknowledgement through `bull job`;
-- legacy repeat scheduler creation, lookup, and removal;
+- `bullmq cmd` add/run behavior;
+- manual `bullmq run` acknowledgement through `bullmq job`;
+- native Job Scheduler creation, lookup, and removal;
 - delayed-job commands;
 - priority listing and counts;
 - global rate-limit commands;
-- deduplication commands and `bull events` delivery;
-- `bull flow` parent/child FlowProducer output.
+- every `msg.cmd` and `bullmq job` action, derived from their runtime switches;
+- deduplication commands and `bullmq events` delivery;
+- `bullmq flow` parent/child and bulk FlowProducer output;
+- worker and producer recovery after Redis restarts;
+- graceful Node-RED shutdown while a manual job is active;
+- one-off delayed notification series and cron notification schedulers.
+
+One test is explicitly Redis-only and says so in its name: the restart check, which stops and restarts the store process on the same port and waits on raw ioredis `ready` events. The PostgreSQL fixture is a container on a Docker-assigned port with no raw client to observe.
 
 ## Playwright
 
@@ -54,7 +65,8 @@ Use Docker for the topology matrix:
 npm run test:deployments
 ```
 
-The runner starts each fixture, waits for Redis readiness, runs `test/integration-deployment.test.js`, and removes volumes between deployments.
+The runner starts each fixture, waits for Redis or PostgreSQL readiness, runs
+`test/integration-deployment.test.js`, and removes volumes between deployments.
 
 Current executable fixtures:
 
@@ -65,8 +77,10 @@ Current executable fixtures:
 - `cluster-tls`: two-node Redis Cluster with TLS and BullMQ `{bull}` prefix coverage
 - `sentinel-auth`: Redis master, two replicas, and three Sentinels with data-node ACL auth
 - `sentinel-tls`: Redis master, two replicas, and three TLS-enabled Sentinels
+- `postgres-plain`: PostgreSQL without TLS
+- `postgres-tls`: PostgreSQL with the Docker-only self-signed TLS fixture
 
-The shared deployment test proves Node-RED load, connection, add/run delivery, required `basecasts` scheduler creation/removal, and absolute scheduler minute/second metadata. TLS fixtures use local self-signed test certificates and disable certificate verification for those Docker-only deployments. MemoryDB remains the certificate-verified TLS deployment path.
+The shared deployment test proves Node-RED load, credential-backed auth, add/run delivery, native scheduler creation/removal, absolute scheduler minute/second metadata, live cancellation retries and `cancelAllJobs`, and OpenTelemetry add/process spans plus completed/duration metrics. The Docker topology fixtures use local self-signed test certificates and disable certificate verification. The dedicated PostgreSQL TLS integration test verifies that certificate through the credential-backed CA field; MemoryDB covers verified TLS for Redis.
 
 ## AWS MemoryDB
 

package/docs/TROUBLESHOOTING.md

@@ -2,8 +2,8 @@
 
 ## Worker Does Not Receive Jobs
 
-- Confirm `bull run` uses the same `bull-queue-server` as `bull cmd`.
-- Confirm Redis is reachable from the Node-RED process.
+- Confirm `bullmq run` uses the same `bullmq-queue-server` as `bullmq cmd`.
+- Confirm the backend (Redis or PostgreSQL) is reachable from the Node-RED process.
 - Confirm the queue name is correct.
 - For scheduled jobs, BullMQ creates the next delayed job only as the previous scheduled job starts processing.
 
@@ -15,14 +15,53 @@ Use a BullMQ prefix with a Redis Cluster hash tag, such as `{bull}`. This keeps
 
 MemoryDB is a Cluster deployment and normally requires TLS from an EC2/VPC client that can reach the endpoint. Use Cluster mode, TLS, ACL username/password, and a reachable VPC network path.
 
+## `Cannot find module 'pg'`
+
+The PostgreSQL backend needs the optional peer dependency: `npm install pg` in the Node-RED user directory, then redeploy. BullMQ loads `pg` while constructing the queue, so this is reported on first use rather than at load, once per config node.
+
+## `PostgreSQL schema "bullmq" is not initialized`
+
+The database has no BullMQ schema and this node was told not to create one. Either enable **Migrations** on the queue config node so it initialises the schema, or run BullMQ's PostgreSQL migrations against the database yourself before deploying. The node stays usable and reports the error rather than hanging.
+
+## PostgreSQL Schema Version Mismatch
+
+The schema was created by a different BullMQ major version. Upgrade this package to a release using the required BullMQ major; PostgreSQL schema downgrades are not supported.
+
+## Unsupported PostgreSQL Version
+
+BullMQ requires PostgreSQL 13 or newer, and recommends 14+. The server version is checked on connect, so this is reported before any job work rather than discovered mid-job.
+
+## PostgreSQL Connections Exhausted
+
+PostgreSQL has a server-wide `max_connections` ceiling, commonly 100, shared with every other client. There is one pool per BullMQ resource rather than one per config node; Worker and QueueEvents also hold one dedicated `LISTEN` connection each. A config node feeding `bullmq cmd`, `bullmq run`, `bullmq events`, and `bullmq flow` therefore costs up to 4 × **Pool Max** + 2 connections — 10 at the default — and many queues or many Node-RED instances multiply that. Raise `max_connections`, lower **Pool Max**, or put a pooler in front.
+
+## PostgreSQL `timeout exceeded when trying to connect`
+
+Two different causes share this message, because node-postgres uses one timeout for both. Either the server is unreachable, or the pool is full and the caller waited 10 seconds for a free connection. Suspect contention when the database is healthy. Raise **Pool Max** only after observing sustained pool waits; worker **Concurrency** does not require a one-to-one pool size because each operation releases its client. Re-check the total against `max_connections` after any increase.
+
+## PostgreSQL Events Table Keeps Growing
+
+Expected, and not fixable from here: BullMQ's PostgreSQL adapter ignores the `maxEvents` trim argument, so event rows accumulate for as long as the queue is used. Prune the table out of band if a flow relies on `bullmq events`.
+
 ## TLS Certificate Errors
 
-Keep TLS verification enabled when possible. Provide the CA certificate or server name needed by the Redis deployment. Disable verification only for controlled deployments that cannot be configured with a trusted CA.
+Keep TLS verification enabled when possible. Provide the CA certificate or server name the deployment needs. On PostgreSQL, node-postgres takes the TLS server name from the connection host when that host is a hostname, so the server-name override only applies when the host is a literal IP address. Disable verification only for controlled deployments that cannot be configured with a trusted CA.
 
-## Repeat Job Is Not Found
+## Job Scheduler Is Not Found
 
-Legacy repeat lookup uses exact scheduler ids. Use `msg.schedulerId`, `msg.jobopts.jobId`, `msg.jobid`, or `msg.jobId` consistently.
+Job Scheduler lookup uses exact ids. Pass the id used during creation in `msg.schedulerId`.
 
 ## Bull v4 Queue Data Missing After Upgrade
 
 Bull v4 Redis data is not automatically migrated. Drain or retire old queues before switching production flows to BullMQ.
+
+## Repeat Job Fires At The Wrong Hour
+
+BullMQ v6 uses `msg.repeat.pattern` and optional `msg.repeat.tz` with `upsertJobScheduler`. Confirm the IANA timezone name and remove/recreate any scheduler created with the wrong timezone.
+
+## `cancelJob` Says No Cancellable Processor
+
+`cancelJob`/`cancelAllJobs` need the acknowledgement behind `msg.bull.ackId`, which only exists for a manual-mode job that has not yet settled. Two different failures look similar:
+
+- No usable `ackId` at all — an immediate-mode job's output message never carries one, and reusing an `ackId` after its job already completed, failed, timed out, or was already cancelled fails the same way. This raises a missing/stale/already-settled acknowledgement error, not "no cancellable processor".
+- `BullMQ found no cancellable processor for job <id>` — the `ackId` was still valid, but BullMQ was no longer tracking a cancellation signal for that job. BullMQ stops tracking a job the moment its processor promise settles, so this is a narrow race: the job completed, failed, or lost its lock between the worker sending the message and the cancel arriving. It is also what you would see if `bullmq run`'s processor were ever changed to take fewer than three parameters, because BullMQ only creates the per-job `AbortController` when the processor declares the signal argument.

package/examples/README.md

@@ -4,32 +4,60 @@ Import these files from the Node-RED editor with **Import > Clipboard**.
 
 ## `example_flow.json`
 
-End-to-end compatibility flow for the `basecasts` queue. It includes the required scheduled job payload and cron expression, plus basic events, manual acknowledgement, and flow examples.
+End-to-end BullMQ v6 flow for the `basecasts` queue. It includes a native Job Scheduler pattern, plus basic events, manual acknowledgement, and flow examples.
 
 ## `bullmq_features.json`
 
 Small BullMQ feature examples that all use one local Redis queue config named `bullmq-features`.
 
 - Delay: `delay: send later` sets `msg.jobopts.delay = 10000`.
+- Delay batch: `delay: series of one-off jobs` sends `addBulk` jobs with increasing delays.
+- Date-time batch: `delay: series at exact date-times` converts explicit ISO-8601 timestamps to delays. Replace the sample 2030 timestamps before running it.
 - Priority: `priority: high priority` sets `msg.jobopts.priority = 1`.
 - Deduplication: `dedupe: same job once` sets `msg.jobopts.deduplication.id` from `msg.payload`.
 - Rate limit: `rate limit: 2 per second` sends `msg.cmd = "setGlobalRateLimit"` with `{ "max": 2, "duration": 1000 }`.
-- Scheduler: `scheduler: every minute` adds a repeatable scheduler with `repeat.pattern`.
-- Events: the `bull events` node emits completed, failed, delayed, deduplicated, duplicated, and progress events.
-- Manual acknowledgement: `manual ack worker` sends a job through `bull job` progress and complete actions.
-- Flow: `flow: parent plus child` sends a parent/child tree to `bull flow`.
+- Scheduler: `scheduler: every minute` uses `upsertJobScheduler` with `msg.repeat.pattern`.
+- Events: the `bullmq events` node emits completed, failed, delayed, deduplicated, duplicated, and progress events.
+- Manual acknowledgement: `manual ack worker` sends a job through `bullmq job` progress and complete actions.
+- Cancel: `cancel: stop running job` demonstrates BullMQ v6 cooperative cancellation on its own `bullmq-cancel` queue, with its own manual-mode worker. It is deliberately separate from the manual ack demo: a `cancelJob` node sharing that worker would cancel the very job the acknowledgement demo is completing.
+- Metrics: `metrics: exportPrometheusMetrics` sends `msg.cmd = "exportPrometheusMetrics"` and returns a Prometheus-formatted string.
+- Flow: `flow: parent plus child` sends a parent/child tree to `bullmq flow`.
 
 Point `bullmq-features` at your Redis deployment before deploying the flow.
 
+## `postgres_backend.json`
+
+The same add/run flow on the PostgreSQL backend, for the `pgcasts` queue. It exists to show what changes when the backend does: only the queue config node.
+
+- The config node sets `backend` to `postgres` with `database`, `username`, `schema`, `max` and `migrate`, and leaves the Redis-only fields (`deployment`, `db`, `clusterNodes`, `sentinels`, `prefix`) empty so switching the selector back to Redis needs no cleanup.
+- `add order job` and `handle order` use exactly the same `msg.cmd`, job options and worker message shape as the Redis examples. Nothing in a producer or worker flow is backend-specific.
+- Install `pg` first (`npm install pg`). It is an optional peer dependency, so a missing install is reported on first use with the command to run.
+- Set the password on the config node before deploying; a shipped flow carries no credential.
+
+## `scheduled_notifications.json`
+
+Scheduling a series of user notifications on the `notifycasts` queue, and delivering each one to a worker. Both halves share one queue and one worker.
+
+- `notify: schedule user series` builds one delayed job per notification from a per-user schedule of explicit ISO-8601 instants, and enqueues the whole series with a single `addBulk`. BullMQ delays in milliseconds rather than at an absolute time, so each instant is converted to a delay at enqueue time; an instant already in the past becomes a delay of `0` and is delivered immediately. Send your own schedule as `msg.payload` in the same shape -- the built-in sample dates are placeholders.
+- Each job gets a `jobId` derived from the user, the notification type and the scheduled instant, so re-sending the same schedule cannot double-book a notification. The instant is used as epoch milliseconds and the parts joined with `-`, because BullMQ rejects a custom job id containing `:`.
+- `notify: cron daily digest` sends `upsertJobScheduler` with a six-field cron `msg.repeat.pattern` (the leading field is seconds) and `msg.repeat.tz`, carrying the notification in `msg.template.data` so every generated job arrives with it.
+- `notify: remove cron digest` sends `removeJobScheduler`. A Job Scheduler outlives a redeploy, so an example that creates one needs a way to remove it.
+- `notification worker` is an immediate-mode `bullmq run` node; `deliver notification` reads the notification out of `msg.payload.message` and sets `msg.topic` to `userId/type`.
+
+A delayed job's time is when it becomes _eligible_, not a guaranteed start: worker availability and concurrency still decide when it actually runs.
+
+Point `notifycasts` at your Redis or PostgreSQL deployment before deploying the flow.
+
 ## `repeatable_jobs.json`
 
-Dedicated repeatable job commands for the `basecasts` queue.
+Native BullMQ v6 Job Scheduler commands for the `basecasts` queue.
 
-- `repeat: add basecasts job` sends `msg.cmd = "add"` with `msg.jobopts.repeat.cron`.
-- `repeat: getRepeatableJobs` lists repeatable schedulers.
-- `repeat: count` counts repeatable schedulers.
-- `repeat: getRepeatableJobByKey` reads the scheduler id from `msg.payload` into `msg.jobid`.
-- `repeat: removeRepeatableByKey` removes the scheduler id from `msg.payload`.
-- `repeat: stopAndRemoveAllJobs` sends `msg.cmd = "stopAndRemoveAllJobs"` to remove schedulers and clean inactive jobs.
+- `scheduler: upsert basecasts job` sends `upsertJobScheduler` with `msg.schedulerId`, `msg.repeat.pattern`, `msg.repeat.tz`, and `msg.template`.
+- `scheduler: upsert with timezone` demonstrates an IANA timezone in `msg.repeat.tz`.
+- `scheduler: getJobSchedulers` lists Job Schedulers.
+- `scheduler: getJobSchedulersCount` counts Job Schedulers.
+- `scheduler: getJobScheduler` reads the exact scheduler id from `msg.payload` into `msg.schedulerId`.
+- `scheduler: removeJobScheduler` removes that exact scheduler id.
+- `scheduler: stopAndRemoveAllJobs` removes schedulers and cleans inactive jobs.
 
 Use the add inject first, then inspect with get/count/get-by-key. Use remove-by-key for one scheduler or stopAndRemoveAllJobs for full cleanup.