@pauldeng/node-red-contrib-bullmq 1.0.0

package/docs/CONNECTIONS.md

@@ -0,0 +1,54 @@
+# Connection Guide
+
+## Standalone Redis
+
+Use deployment `single`, host, port, optional database, optional username/password, and optional TLS.
+
+Producer commands use bounded retries so Node-RED input handlers fail instead of hanging forever. Workers and QueueEvents use persistent retry behavior required by BullMQ.
+
+## Redis Cluster
+
+Use deployment `cluster` and provide startup nodes as comma or newline separated `host:port` values.
+
+Cluster auth and TLS are applied through ioredis `redisOptions`. The runtime sets a DNS lookup passthrough for TLS-enabled cluster discovery.
+
+Use a BullMQ prefix with a hash tag, normally `{bull}`.
+
+## AWS MemoryDB
+
+Use deployment `cluster`.
+
+Typical settings:
+
+- cluster endpoint and port as a startup node;
+- ACL username and password;
+- TLS enabled;
+- prefix `{bull}`;
+- client located in a VPC/network path that can reach MemoryDB.
+
+Do not write MemoryDB credentials into flows, examples, docs, or logs.
+
+## Sentinel
+
+Use deployment `sentinel` and configure:
+
+- Sentinel endpoints;
+- master name;
+- optional Redis data-node username/password;
+- optional Sentinel username/password;
+- optional TLS for Redis data nodes;
+- optional TLS for Sentinel discovery.
+
+Sentinel authentication is separate from Redis data-node authentication.
+
+## TLS
+
+TLS options:
+
+- verify unauthorized certificates by default;
+- optional CA;
+- optional client certificate;
+- optional client private key;
+- optional server name.
+
+Disable verification only when the Redis deployment cannot be configured with a trusted CA and the risk is understood.

package/docs/MIGRATION.md

@@ -0,0 +1,37 @@
+# Migration From Bull v4
+
+## What Stays Compatible
+
+- Node types `bull-queue-server`, `bull cmd`, and `bull run`.
+- Message-driven `msg.cmd` command dispatch.
+- `msg.payload` compatibility for added jobs and worker output.
+- `msg.jobopts.repeat.cron` for scheduled jobs.
+
+## What Changes
+
+- Runtime dependency is BullMQ 5.78.0.
+- `bull` and `sprintf-js` are removed.
+- Repeatable jobs use BullMQ Job Schedulers.
+- Scheduled jobs require a stable scheduler id.
+- `bull run` uses BullMQ Worker instead of Bull v4 `queue.process`.
+
+## Repeat Jobs
+
+Legacy:
+
+```js
+msg.jobopts = {
+  jobId: msg.payload,
+  repeat: { cron: "30 9,19,29,39,49,59 * * * *" },
+};
+```
+
+Runtime translation:
+
+- scheduler id: `msg.schedulerId` or `msg.jobopts.jobId`;
+- repeat pattern: `msg.jobopts.repeat.pattern`;
+- template data: `msg.jobData` or `{ payload: msg.payload }`.
+
+## Data Migration
+
+Bull v4 and BullMQ do not provide a supported Redis data migration contract. Do not assume existing delayed, waiting, active, completed, or repeatable Bull v4 keys will be usable by BullMQ.

package/docs/NODE_GUIDE.md

@@ -0,0 +1,83 @@
+# Node Guide
+
+## `bull-queue-server`
+
+Configures queue name, Redis deployment, credentials, TLS, and BullMQ prefix.
+
+Deployment modes:
+
+- `single`: standalone Redis.
+- `cluster`: Redis Cluster and AWS MemoryDB.
+- `sentinel`: Redis Sentinel.
+
+Use `{bull}` for Cluster and MemoryDB unless you have a tested prefix strategy.
+
+## `bull cmd`
+
+Input node for producer and administration commands.
+
+Input:
+
+- `msg.cmd`: command name. Defaults to `add`.
+- `msg.payload`: compatibility payload.
+- `msg.jobData`: full BullMQ job data when supplied.
+- `msg.jobName`: BullMQ job name. Defaults to `default`.
+- `msg.jobopts`: BullMQ job options.
+
+Output:
+
+- successful result in `msg.payload`;
+- errors go to `done(err)` or `node.error(err, msg)`.
+
+## `bull run`
+
+Worker node with no input and one output.
+
+Output message:
+
+- `msg.payload`: `job.data.payload` when present, otherwise full `job.data`;
+- `msg.job`: serialized job metadata;
+- `msg.bull`: queue and job context.
+
+Completion modes:
+
+- `immediate`: complete after sending the message.
+- `manual`: wait for downstream `bull job` acknowledgement. Fails the job after the ack timeout; set the timeout to `0` to wait indefinitely.
+
+## `bull job`
+
+Acts on manual-mode active jobs. Actions can be configured or supplied in `msg.cmd`.
+
+Terminal actions:
+
+- `complete`
+- `fail`
+- `failUnrecoverable`
+- `rateLimit`
+
+Non-terminal actions:
+
+- `progress`
+- `removeDeduplicationKey`
+- `getChildrenValues`
+- `getFailedChildrenValues`
+- `removeUnprocessedChildren`
+
+## `bull events`
+
+QueueEvents source node. Empty event filter subscribes to the default documented event list.
+
+Output:
+
+- `msg.topic`: event name;
+- `msg.payload`: BullMQ event payload;
+- `msg.bull`: queue, event, and event id metadata.
+
+## `bull flow`
+
+Adds a BullMQ FlowProducer tree.
+
+Input:
+
+- `msg.payload`: BullMQ flow tree;
+- `msg.flowopts`: optional FlowProducer options.

package/docs/REFERENCE_MAP.md

@@ -0,0 +1,51 @@
+# Reference Map
+
+## Runtime
+
+- `bull-queue.js`: Node-RED registration and runtime glue.
+- `lib/connections.js`: Redis deployment normalization and ioredis descriptors.
+- `lib/scheduler.js`: legacy repeat-cron to BullMQ Job Scheduler normalization.
+- `lib/commands.js`: `bull cmd` dispatch.
+- `lib/serialization.js`: message-safe BullMQ serialization.
+- `lib/acknowledgements.js`: manual-acknowledgement registry for `bull run`/`bull job`. Entries self-remove on settle.
+
+## Editor
+
+- `bull-queue.html`: Node-RED edit dialogs and help text.
+- `icons/bull_icon.png`: palette icon.
+
+## Tests
+
+- `test/package-contract.test.js`: dependency and runtime import contract.
+- `test/connections.test.js`: Redis topology option normalization.
+- `test/scheduler.test.js`: repeat scheduler compatibility.
+- `test/commands.test.js`: command dispatch behavior.
+- `test/acknowledgements.test.js`: manual-acknowledgement registry lifecycle and leak prevention.
+- `test/node-red-registration.test.js`: Node-RED node type registration.
+- `test/editor-contract.test.js`: static editor surface.
+- `test/docs-contract.test.js`: required docs and examples.
+- `test/docker-matrix-contract.test.js`: Docker deployment fixture and runner contract.
+- `test/integration-standalone.test.js`: opt-in local Redis Node-RED runtime flow tests for add/run, manual acknowledgement, repeat schedulers, delayed jobs, priorities, rate limits, deduplication events, and flow producer jobs.
+- `test/integration-deployment.test.js`: opt-in external Redis deployment flow test used by Docker and MemoryDB.
+
+## Deployment Test Fixtures
+
+- `scripts/run-deployment-tests.js`: Docker and optional MemoryDB deployment runner.
+- `test/deployments/single-noauth/`: standalone Redis without auth.
+- `test/deployments/single-auth/`: standalone Redis with ACL auth.
+- `test/deployments/single-tls/`: standalone Redis with TLS.
+- `test/deployments/cluster-auth/`: Redis Cluster with ACL auth.
+- `test/deployments/cluster-tls/`: Redis Cluster with TLS.
+- `test/deployments/sentinel-auth/`: Redis Sentinel with data-node ACL auth.
+- `test/deployments/sentinel-tls/`: Redis Sentinel with TLS for data-node and Sentinel connections.
+- `test/deployments/tls-certs/`: local self-signed certificates for Docker TLS fixtures.
+
+## User Docs
+
+- `README.md`: overview and installation.
+- `docs/NODE_GUIDE.md`: node behavior.
+- `docs/COMMANDS.md`: command reference.
+- `docs/CONNECTIONS.md`: Redis deployments.
+- `docs/MIGRATION.md`: Bull v4 migration.
+- `docs/TESTING.md`: verification plan.
+- `docs/TROUBLESHOOTING.md`: operational issues.

package/docs/RELEASE.md

@@ -0,0 +1,108 @@
+# Release Guide
+
+Checklist for publishing `@pauldeng/node-red-contrib-bullmq` to npm and the
+Node-RED Flow Library, following current (2025-2026) supply-chain practice.
+
+Publishing uses **npm trusted publishing (OIDC)** from GitHub Actions. There is
+no long-lived `NPM_TOKEN`: the release workflow mints a short-lived OIDC
+credential per run and attaches build provenance automatically. (Classic npm
+tokens were revoked in December 2025; only granular tokens remain, and they are
+used here only for the one-time first publish.)
+
+## 1. Preflight
+
+1. Confirm `package.json` has the intended `version` and the name `@pauldeng/node-red-contrib-bullmq`.
+2. Confirm the GitHub repository is `https://github.com/pauldeng/node-red-contrib-bullmq`.
+3. Confirm BullMQ is pinned to exactly `5.78.0`.
+4. Confirm no examples, docs, fixtures, or logs contain Redis, Sentinel, or MemoryDB secrets.
+5. Update `CHANGELOG.md` for the new version.
+
+## 2. Local verification
+
+```sh
+npm ci
+npm test                                  # unit + contract tests
+npm run test:playwright                   # editor tests
+npm run format:check                      # prettier gate
+npm run validate                          # Node-RED scorecard preflight (node-red-dev)
+npm audit --omit=dev --audit-level=moderate
+npm pack --dry-run                        # confirm the tarball is clean
+```
+
+For deployment changes, also run:
+
+```sh
+npm run test:deployments
+```
+
+MemoryDB verification is opt-in and environment-only:
+
+```sh
+MEMORYDB_ENABLED=1 npm run test:deployments
+```
+
+## 3. One-time account and repository setup
+
+Do these once, before the first release.
+
+### npm account
+
+- [ ] Enable 2FA for authorization and writes: `npm profile enable-2fa auth-and-writes`.
+- [ ] Make sure the `@pauldeng` scope exists and you can publish under it.
+- [ ] Create a short-lived **granular** access token (write scope, this package only) for the one-time first publish in step 4.
+
+### GitHub repository settings
+
+- [ ] **Branch ruleset** on `master` (Settings -> Rules -> Rulesets): require a pull request with at least one review, require the CI status checks to pass, block force pushes, restrict deletions. Optionally require linear history and signed commits.
+- [ ] **Code security and analysis** (Settings): enable Dependabot alerts, Dependabot security updates, secret scanning, push protection, and private vulnerability reporting.
+- [ ] **Workflow permissions** (Settings -> Actions -> General): set the default `GITHUB_TOKEN` to read-only.
+- [ ] **CodeQL**: either rely on `.github/workflows/codeql.yml`, or enable Code scanning "default setup" in the Security tab.
+- [ ] **`release` environment** (Settings -> Environments -> New environment -> `release`): add yourself as a required reviewer so `publish.yml` waits for manual approval.
+
+## 4. First publish (one time, manual)
+
+Trusted publishing can only be configured on a package that already exists, so
+publish v1.0.0 manually first.
+
+```sh
+npm publish --access public      # authenticated with the granular token + 2FA
+```
+
+After publishing, inspect the npm package page and verify the README, badges,
+license, repository link, and tarball contents.
+
+## 5. Configure trusted publishing
+
+On npmjs.com -> the package -> Settings -> Trusted Publisher:
+
+- [ ] Add a GitHub Actions publisher: owner `pauldeng`, repository `node-red-contrib-bullmq`, workflow filename `publish.yml` (case-sensitive), optional environment `release`.
+- [ ] Set the package to "Require two-factor authentication and disallow tokens".
+- [ ] Revoke the temporary granular token from step 3.
+
+## 6. Subsequent releases (automated)
+
+For every release after the first:
+
+1. Bump the version and tag it (a signed tag is recommended):
+   ```sh
+   npm version <patch|minor|major>
+   git push --follow-tags
+   ```
+2. Create a **GitHub Release** for the new tag. Publishing the `release` triggers `.github/workflows/publish.yml`.
+3. Approve the `release` environment deployment when prompted.
+4. The workflow publishes to npm over OIDC with provenance - no token needed.
+
+Confirm `npm view @pauldeng/node-red-contrib-bullmq` shows the new version and
+that the npm page displays the green provenance badge.
+
+## 7. Node-RED Flow Library
+
+The Flow Library indexes npm but submission is **manual** (since April 2020).
+Scoped packages are fully supported and listed.
+
+- [ ] First time: submit via the `+` button at <https://flows.nodered.org/add/node> using package `@pauldeng/node-red-contrib-bullmq`.
+- [ ] For later versions: use the "request refresh" link on the package's library page (visible when logged in).
+
+## 8. Optional
+
+- [ ] Self-certify the OpenSSF Best Practices badge at <https://www.bestpractices.dev>.

package/docs/TESTING.md

@@ -0,0 +1,88 @@
+# Testing
+
+## Fast Local Tests
+
+```sh
+npm test
+```
+
+This runs built-in `node:test` suites for package metadata, connection normalization, scheduler compatibility, command dispatch, editor surface, registration, Docker fixture contracts, docs, and examples.
+
+## Node-RED Runtime Tests
+
+Use `node-red-node-test-helper` for flow-level tests. Runtime coverage should load actual Node-RED flows for:
+
+- `bull cmd` success and failure paths;
+- `bull run` immediate and manual modes;
+- `bull job` acknowledgement actions;
+- `bull events`;
+- `bull flow`;
+- legacy flow compatibility.
+
+The current standalone Redis integration suite is opt-in:
+
+```sh
+npm run test:integration
+```
+
+It starts a temporary local `redis-server`, loads real Node-RED flows, and verifies:
+
+- `bull cmd` add/run behavior;
+- manual `bull run` acknowledgement through `bull job`;
+- legacy repeat scheduler creation, lookup, and removal;
+- delayed-job commands;
+- priority listing and counts;
+- global rate-limit commands;
+- deduplication commands and `bull events` delivery;
+- `bull flow` parent/child FlowProducer output.
+
+## Playwright
+
+Editor changes require Playwright coverage:
+
+```sh
+npm run test:playwright
+```
+
+Cover deployment field visibility, credential persistence, TLS fields, and edit dialogs for all node types.
+
+## Docker Matrix
+
+Use Docker for the topology matrix:
+
+```sh
+npm run test:deployments
+```
+
+The runner starts each fixture, waits for Redis readiness, runs `test/integration-deployment.test.js`, and removes volumes between deployments.
+
+Current executable fixtures:
+
+- `single-noauth`: standalone Redis without auth or TLS
+- `single-auth`: standalone Redis with ACL auth
+- `single-tls`: standalone Redis with TLS
+- `cluster-auth`: two-node Redis Cluster with ACL auth and BullMQ `{bull}` prefix coverage
+- `cluster-tls`: two-node Redis Cluster with TLS and BullMQ `{bull}` prefix coverage
+- `sentinel-auth`: Redis master, two replicas, and three Sentinels with data-node ACL auth
+- `sentinel-tls`: Redis master, two replicas, and three TLS-enabled Sentinels
+
+The shared deployment test proves Node-RED load, connection, add/run delivery, required `basecasts` scheduler creation/removal, and absolute scheduler minute/second metadata. TLS fixtures use local self-signed test certificates and disable certificate verification for those Docker-only deployments. MemoryDB remains the certificate-verified TLS deployment path.
+
+## AWS MemoryDB
+
+MemoryDB tests are opt-in. Use environment variables only:
+
+- `MEMORYDB_ENDPOINT`
+- `MEMORYDB_PORT`
+- `MEMORYDB_USERNAME`
+- `MEMORYDB_PASSWORD`
+- optional `MEMORYDB_TLS`
+- explicit enable flag such as `MEMORYDB_ENABLED=1`
+
+Run MemoryDB with the same deployment harness:
+
+```sh
+MEMORYDB_ENABLED=1 npm run test:deployments
+```
+
+Never write these values to repository files, examples, fixtures, snapshots, or logs.

package/docs/TROUBLESHOOTING.md

@@ -0,0 +1,28 @@
+# Troubleshooting
+
+## Worker Does Not Receive Jobs
+
+- Confirm `bull run` uses the same `bull-queue-server` as `bull cmd`.
+- Confirm Redis is reachable from the Node-RED process.
+- Confirm the queue name is correct.
+- For scheduled jobs, BullMQ creates the next delayed job only as the previous scheduled job starts processing.
+
+## Cluster `CROSSSLOT` Errors
+
+Use a BullMQ prefix with a Redis Cluster hash tag, such as `{bull}`. This keeps BullMQ queue keys in the same slot for atomic operations.
+
+## MemoryDB Connection Hangs
+
+MemoryDB is a Cluster deployment and normally requires TLS from an EC2/VPC client that can reach the endpoint. Use Cluster mode, TLS, ACL username/password, and a reachable VPC network path.
+
+## TLS Certificate Errors
+
+Keep TLS verification enabled when possible. Provide the CA certificate or server name needed by the Redis deployment. Disable verification only for controlled deployments that cannot be configured with a trusted CA.
+
+## Repeat Job Is Not Found
+
+Legacy repeat lookup uses exact scheduler ids. Use `msg.schedulerId`, `msg.jobopts.jobId`, `msg.jobid`, or `msg.jobId` consistently.
+
+## Bull v4 Queue Data Missing After Upgrade
+
+Bull v4 Redis data is not automatically migrated. Drain or retire old queues before switching production flows to BullMQ.

package/examples/README.md

@@ -0,0 +1,35 @@
+# Examples
+
+Import these files from the Node-RED editor with **Import > Clipboard**.
+
+## `example_flow.json`
+
+End-to-end compatibility flow for the `basecasts` queue. It includes the required scheduled job payload and cron expression, plus basic events, manual acknowledgement, and flow examples.
+
+## `bullmq_features.json`
+
+Small BullMQ feature examples that all use one local Redis queue config named `bullmq-features`.
+
+- Delay: `delay: send later` sets `msg.jobopts.delay = 10000`.
+- Priority: `priority: high priority` sets `msg.jobopts.priority = 1`.
+- Deduplication: `dedupe: same job once` sets `msg.jobopts.deduplication.id` from `msg.payload`.
+- Rate limit: `rate limit: 2 per second` sends `msg.cmd = "setGlobalRateLimit"` with `{ "max": 2, "duration": 1000 }`.
+- Scheduler: `scheduler: every minute` adds a repeatable scheduler with `repeat.pattern`.
+- Events: the `bull events` node emits completed, failed, delayed, deduplicated, duplicated, and progress events.
+- Manual acknowledgement: `manual ack worker` sends a job through `bull job` progress and complete actions.
+- Flow: `flow: parent plus child` sends a parent/child tree to `bull flow`.
+
+Point `bullmq-features` at your Redis deployment before deploying the flow.
+
+## `repeatable_jobs.json`
+
+Dedicated repeatable job commands for the `basecasts` queue.
+
+- `repeat: add basecasts job` sends `msg.cmd = "add"` with `msg.jobopts.repeat.cron`.
+- `repeat: getRepeatableJobs` lists repeatable schedulers.
+- `repeat: count` counts repeatable schedulers.
+- `repeat: getRepeatableJobByKey` reads the scheduler id from `msg.payload` into `msg.jobid`.
+- `repeat: removeRepeatableByKey` removes the scheduler id from `msg.payload`.
+- `repeat: stopAndRemoveAllJobs` sends `msg.cmd = "stopAndRemoveAllJobs"` to remove schedulers and clean inactive jobs.
+
+Use the add inject first, then inspect with get/count/get-by-key. Use remove-by-key for one scheduler or stopAndRemoveAllJobs for full cleanup.