mail-time 4.1.0 → 5.1.0

package/README.md

@@ -19,7 +19,7 @@ Bulletproof email queue for [horizontally scaled](#sending-emails-from-a-cluster
 
 `MailTime` runs in one of two modes:
 
-- **`server`** — drains the queue and sends emails via SMTP. The cluster-aware lease guarantees exactly one server sends each email even when many are running.
+- **`server`** — drains the queue and sends emails via SMTP. Per-row leases let one worker claim each attempt even when many are running.
 - **`client`** — only enqueues emails. Use for app servers in a "dedicated mail micro-service" topology.
 
 Many clients + one or more servers coexist behind the same `prefix` in the same store.
@@ -29,7 +29,7 @@ Many clients + one or more servers coexist behind the same `prefix` in the same
 - 🏢 **Horizontally scaled** — synchronize one queue across N processes/hosts/DCs.
 - 🔁 **Multi-SMTP rotation** — `backup` (failover) and `balancer` (round-robin) strategies.
 - 💪 **Built-in retries** — storage-backed retry with per-letter transport pinning.
-- 🎯 **Per-recipient retries** — when a multi-`to` send is partially rejected, only the un-accepted addresses are retried; delivered ones never see a duplicate.
+- 🎯 **Per-recipient retries** — when a multi-`to` send is partially rejected, later attempts exclude accepted addresses.
 - 📮 **Email concatenation** — fold same-`to` emails arriving inside a window into one letter.
 - 🎛️ **One-line setup** — built-in [presets](#settings-presets) for `transactional`, `otp`, `newsletter`, `marketing`, `notifications`, `alerts`.
 - 🛢️ **Three first-party storages** — MongoDB, Redis, PostgreSQL. Plus a [custom-adapter contract](https://github.com/veliovgroup/mail-time/blob/master/docs/queue-api.md).
@@ -84,7 +84,7 @@ MailTime keeps every letter in the queue until SMTP confirms delivery.
 
 ### Sending emails from a cluster
 
-Most apps schedule recurring emails (daily digest, weekly summary). On a single server this is trivial. In a cluster, every node would otherwise send the same email N times. MailTime's lease prevents the duplicate sends.
+Most apps schedule recurring emails (daily digest, weekly summary). On a single server this is trivial. In a cluster, every node would otherwise send the same email N times. MailTime's lease prevents concurrent duplicate claims. SMTP acceptance and queue completion cannot be one atomic transaction: a worker crash between them can still cause a later retry. See [reliability boundary](https://github.com/veliovgroup/mail-time/blob/master/docs/tuning.md#reliability-boundary).
 
 ```ascii
 |===================THE=CLUSTER===================| |=QUEUE=|
@@ -271,9 +271,7 @@ process.on('SIGTERM', async () => {
 });
 ```
 
-Or explicitly: `mailQueue.destroy(); await mailQueue.drain();`
-
-`destroy()` is idempotent and stops the scheduler. Always call it from tests; pair with `drain()` when iterate-driven sends ran.
+`destroy()` without `{ drain: true }` stops immediately: in-flight completions make no storage writes and their claims recover after `sendingTimeout`. Use it for forced teardown, not graceful shutdown.
 
 #### Pause / resume a server (backpressure)
 
@@ -300,7 +298,7 @@ Queue storage and scheduler storage can be the same store or different ones. Use
 
 | Queue    | Scheduler (`josk`) | Best for                                     |
 | -------- | ------------------ | -------------------------------------------- |
-| Postgres | Postgres           | Multi-DC, mixed clocks, strict exactly-once. |
+| Postgres | Postgres           | Multi-DC, mixed clocks, strongest consistency. |
 | Redis    | Redis              | High-throughput single-region.               |
 | Mongo    | Mongo              | Apps already on Mongo (especially Meteor).   |
 | Mongo    | Redis              | Durable letter storage + sub-second polling. |
@@ -338,11 +336,11 @@ const mailTime = new MailTime(mailTimePreset('otp', {
 | Preset          | Shape                                                                                                                                              | Best for                                                                                |
 | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
 | `transactional` | `retries: 30`, `retryDelay: 10s`, `concatEmails: false`, `concurrency: 1`, `josk.zombieTime: 120s`                                                 | Receipts, password resets, account changes, welcome emails.                             |
-| `otp`           | `retries: 5`, `retryDelay: 2s`, snappy `revolvingInterval: 1024` + jitter `256/1024`, `concurrency: 4`, `sendingTimeout: 60s`                      | Sign-in codes, 2FA, verification codes — stale OTPs aren't worth resending forever.     |
+| `otp`           | `retries: 5`, `retryDelay: 2s`, snappy `revolvingInterval: 1024` + jitter `256/1024`, `concurrency: 4`, `sendingTimeout: 2min`                     | Sign-in codes, 2FA, verification codes — stale OTPs aren't worth resending forever.     |
 | `newsletter`    | `concatEmails: true` with a 5-minute fold window, `concatSubject: 'Your updates'`, `retries: 5`, `retryDelay: 60s`, `concurrency: 2`, `sendingTimeout: 10min`, `josk.zombieTime: 5min` | Scheduled digests, weekly summaries, "what's new" emails.                               |
 | `marketing`     | `retries: 10`, `retryDelay: 30s`, `concatEmails: false`, `concurrency: 5`, `josk.zombieTime: 3min`                                                 | Promotional / campaign blasts where each letter is unique.                              |
 | `notifications` | `concatEmails: true` with a 60-second fold window, `concatSubject: 'New activity'`, `retries: 8`, `retryDelay: 30s`, `concurrency: 3`, `josk.zombieTime: 3min` | App / social activity (likes, mentions, follows) where bursts collapse into one letter. |
-| `alerts`        | `retries: 20`, `retryDelay: 5s`, snappy `revolvingInterval: 1024` + jitter `256/1024`, `concurrency: 2`, `sendingTimeout: 60s`                     | Ops / admin alerts: monitoring, error reports, escalations.                             |
+| `alerts`        | `retries: 20`, `retryDelay: 5s`, snappy `revolvingInterval: 1024` + jitter `256/1024`, `concurrency: 2`, `sendingTimeout: 2min`                     | Ops / admin alerts: monitoring, error reports, escalations.                             |
 
 Presets are equally useful on `type: 'client'` instances — keys that don't apply to the client role are simply ignored.
 
@@ -371,8 +369,12 @@ Defaults fit moderate traffic in a single region. Reach for a [preset](#settings
 | Option                                         | Default          | Change when                                                                                                                        |
 | ---------------------------------------------- | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
 | `mode`                                         | `'batch'`        | `'one'` to claim a single row per tick (fairness over throughput across cluster nodes)                                             |
-| `concurrency`                                  | `1`              | Raise to send N emails in parallel per instance. Bounded by your SMTP / API rate limits. The CAS on `isSending` keeps it safe.     |
-| `sendingTimeout`                               | `300000` (5 min) | Stale-lock recovery window. Must exceed worst-case SMTP roundtrip; lower it only when you're confident sends never take that long. |
+| `concurrency`                                  | `1`              | Raise to send N emails in parallel per instance. CAS blocks concurrent claims; SMTP rate limits still apply.                     |
+| `sendingTimeout`                               | `300000` (5 min) | Stale-lock recovery window. Must exceed worst-case SMTP roundtrip; MailTime logs a warning below `120000`. |
+| `renewClaim`                                   | `sendingTimeout / 3` | Set `false` to go back to a single stamp at claim time. Lower it if your storage round-trip is slow relative to `sendingTimeout`. |
+| `maxRenewals`                                  | `10`             | Renewal-attempt ceiling. Recovery begins `sendingTimeout` after last successful stamp; with responsive storage this is roughly `sendingTimeout + maxRenewals × renewClaim`. |
+| `shouldFailOver`                               | —                | Set when your transport can tell "never delivered" from "may have been delivered" — see [Transport fail-over](#transport-fail-over). |
+| `strictPayload`                                | `false`          | Turn on when anything other than your own app can write to the queue storage. |
 | `revolvingInterval`                            | `1536` ms        | Lower → faster pickup; higher → less scheduler I/O                                                                                 |
 | `josk.minRevolvingDelay` / `maxRevolvingDelay` | `512` / `2048`   | Lower both → snappier polls, more storage load                                                                                     |
 | `josk.zombieTime`                              | `60000`          | **Never below 60s.** Iterate releases the JoSk lease as soon as scanning ends — only a stalled storage scan can blow this.         |
@@ -385,12 +387,54 @@ Defaults fit moderate traffic in a single region. Reach for a [preset](#settings
 
 For deeper JoSk semantics (lease lifecycle, scheduler adapters, recurring tasks), install the JoSk skill: **`npx skills add veliovgroup/josk`**.
 
+## Transport fail-over
+
+`backup` rotates after `failsToNext` failures. Veto rotation when an MTA may already hold the message:
+
+```js
+new MailTime({
+  // ...
+  shouldFailOver: (error) => error?.code !== 'EMESSAGE', // nodemailer: rejected after DATA
+});
+```
+
+Transports may also set `error.mayFailOver = false`. Retry timing stays unchanged. Edge cases: [v4.1 → v5 migration guide](https://github.com/veliovgroup/mail-time/blob/master/docs/migration-v4.1-v5.md#shouldfailovererror-info-email).
+
+## Queue payload trust
+
+`raw` is always refused. Use `strictPayload` when queue writers should not control attachments, URLs, envelope, DKIM, or other nodemailer capabilities:
+
+```js
+new MailTime({
+  // ...
+  strictPayload: true,
+  allowedMailFields: ['attachments'], // optional allowlist extension
+});
+```
+
+`strictPayload` also forces `disableFileAccess` and `disableUrlAccess`. Threat model and built-in allowlist: [v4.1 → v5 migration guide](https://github.com/veliovgroup/mail-time/blob/master/docs/migration-v4.1-v5.md#strictpayload--allowedmailfields).
+
+## Resolving the sender
+
+`from` receives selected transport plus `{ index, from }`:
+
+```js
+new MailTime({
+  // ...
+  from: (transport, details) => `"Acme" <${details.from}>`,
+});
+```
+
+Use `details.from`, not `transport.options.from`, for class-instance transports. `MailTime.transportFrom(transport)` exposes same resolution. Details: [v4.1 → v5 migration guide](https://github.com/veliovgroup/mail-time/blob/master/docs/migration-v4.1-v5.md#fromtransport-details-and-mailtimetransportfrom).
+
 ## Templates
 
 Two Mustache-like placeholder forms:
 
-- `{{key}}` — string interpolation, strips HTML from the value (safe for plain text).
-- `{{{key}}}` — raw HTML interpolation.
+- `{{key}}` — string interpolation. **HTML-escaped** in HTML contexts (`html`, `template`, `concatDelimiter`); verbatim in `text` bodies and subject headers, which are not HTML.
+- `{{{key}}}` — raw interpolation, never escaped. Only use it for values you produced yourself.
+
+Migration behavior: [v4.1 → v5 template escaping](https://github.com/veliovgroup/mail-time/blob/master/docs/migration-v4.1-v5.md#2-key-escapes-html-instead-of-stripping-tags).
 
 Every `sendMail` option is available inside `text`, `html`, and the wrapping template:
 
@@ -441,12 +485,17 @@ await mailQueue.sendMail({
 | `concatDelay`                 | `number` (ms)                                       | `60000`                    | Fold window.                                                                                                                                                                                |
 | `revolvingInterval`           | `number` (ms)                                       | `1536`                     | Queue iteration interval.                                                                                                                                                                   |
 | `mode`                        | `'one' \| 'batch'`                                  | `'batch'`                  | `'batch'` claims every due row per tick; `'one'` claims a single row per tick.                                                                                                              |
-| `concurrency`                 | `number`                                            | `1`                        | Parallel SMTPs per instance. The CAS on `isSending` prevents duplicate delivery.                                                                                                            |
-| `sendingTimeout`              | `number` (ms)                                       | `300000`                   | Window after which a stuck `isSending=true` row becomes eligible again. Must exceed worst-case SMTP roundtrip.                                                                              |
+| `concurrency`                 | `number`                                            | `1`                        | Parallel SMTPs per instance. CAS blocks concurrent claims for one row.                                                                                                                      |
+| `sendingTimeout`              | `number` (ms)                                       | `300000`                   | Window after which a stuck `isSending=true` row becomes eligible again. Must exceed worst-case SMTP roundtrip; values below `120000` log a warning.                                          |
+| `renewClaim`                  | `boolean \| number` (ms)                            | `sendingTimeout / 3`       | Re-stamp `sendingAt` on the claimed row while its SMTP roundtrip runs, so a slow-but-healthy send never loses its lock to a recovery worker. `false` restores the v4 single-stamp behaviour. |
+| `maxRenewals`                 | `number`                                            | `10`                       | Renewal-attempt budget per send. Recovery begins `sendingTimeout` after last successful stamp; slow renewal writes can extend elapsed time beyond the nominal interval calculation.          |
+| `shouldFailOver`              | `(error, info, email) => boolean`                   | —                          | Veto rotating to the next transport for this failure. Default: rotate unless the error carries `mayFailOver === false`. See [Transport fail-over](#transport-fail-over).                     |
+| `strictPayload`               | `boolean`                                           | `false`                    | Narrow every queued letter to `allowedMailFields` and force nodemailer's `disableFileAccess` / `disableUrlAccess`. See [Queue payload trust](#queue-payload-trust).                          |
+| `allowedMailFields`           | `string[]`                                          | —                          | Extra field names permitted under `strictPayload` (added to the built-in allowlist).                                                                                                        |
 | `verifyTransports`            | `boolean`                                           | `true`                     | Probe each transport via `transport.verify()` once at `ready()`. Failing transports are marked unusable, surfaced through `onError(error, null, { transportIndex, phase: 'verify' })`, and skipped during rotation/fallback. Throws from `ready()` if **every** transport fails. Transports without a `verify()` method are treated as healthy. |
 | `template`                    | `string`                                            | `'{{{html}}}'`             | Default envelope.                                                                                                                                                                           |
 | `prefix`                      | `string`                                            | `''`                       | Queue namespace. **Same** on every `client` and `server` for one logical queue; **different** per email class. Inherited by the queue adapter; JoSk scheduler uses `mailTimeQueue<prefix>`. |
-| `from`                        | `string \| (transport) => string`                   | —                          | Strongly recommended for spam-passing `From:` formatting.                                                                                                                                   |
+| `from`                        | `string \| (transport, details) => string`          | —                          | Strongly recommended for spam-passing `From:` formatting. `details` is `{ index, from }` — see [Resolving the sender](#resolving-the-sender).                                                |
 | `debug`                       | `boolean`                                           | `false`                    | Verbose logs.                                                                                                                                                                               |
 | `onSent(email, info)`         | `function`                                          | —                          | Called once the task is fully delivered. `email.mailOptions[i].accepted` lists every address that got through (across all attempts).                                                        |
 | `onError(error, email, info)` | `function`                                          | —                          | Called once the retry budget is exhausted with at least one un-accepted recipient. `email.mailOptions[i].rejected` lists each un-delivered address with its last error. Also fires once per transport that fails `verify()` at startup with `email === null` and `info = { transportIndex, phase: 'verify' }`. |
@@ -461,6 +510,7 @@ await mailQueue.sendMail({
 | `minRevolvingDelay`       | `512`             | Lower bound of poll window.                                                                                    |
 | `maxRevolvingDelay`       | `2048`            | Upper bound.                                                                                                   |
 | `zombieTime`              | `60000`           | Re-claim if `queue.iterate()` runs longer than this. **Do not drop below 60s.**                                |
+| `lockLeaseTime`           | `30000`           | JoSk scheduler lease TTL. JoSk floors it at `2 * maxRevolvingDelay + 1000`; increase for slow storage claim batches. Separate from `zombieTime`, which controls task recovery. |
 | `execute`                 | `'batch'`         | JoSk scheduler batching; low impact for MailTime (one interval task per instance).                             |
 | `concurrency`             | `Infinity`        | Cap overlapping JoSk handler runs on **this** process (`1` if ticks pile up).                                  |
 | `autoClear`               | `false`           | Remove orphan tasks from storage.                                                                              |
@@ -478,21 +528,25 @@ For deeper JoSk semantics, install the JoSk skill: **`npx skills add veliovgroup
 - `cancel(uuid)` — alias of `cancelMail`.
 - `ping()` → `Promise<{status, code, statusCode, paused?, error?}>`. Pings scheduler then queue; `paused` reflects `isPaused`.
 - `ready()` → `Promise<MailTime>`. Awaits all startup work; rejects with `.cause` on storage failure.
-- `destroy(opts?)` → `boolean` or `Promise<boolean>` when `{ drain: true }`. Stops scheduler. Idempotent. Use `destroy({ drain: true })` or `await drain()` after `destroy()` for graceful shutdown.
+- `destroy(opts?)` → `boolean` or `Promise<boolean>` when `{ drain: true }`. Stops scheduler. Idempotent. Use `await destroy({ drain: true })` for graceful shutdown; plain `destroy()` aborts completion writes.
 - `drain()` → `Promise<void>`. Resolves once every in-flight SMTP attempt finishes. Useful in tests and graceful-shutdown paths.
-- `pause()` / `resume()` → `boolean`. Server-only reversible backpressure; no-ops on `client` or after `destroy()`. See [Pause / resume](#pause--resume-a-server-backpressure).
+- `pause()` / `resume()` → `boolean`. Server-only reversible backpressure; no-ops on `client` or after `destroy()`. See [Shutdown](#6-shutdown).
 - `isPaused` → `boolean`. Read-only; always `false` on `client`.
 
 ### Queue constructors
 
 | Constructor                              | Required option                                      | Optional                                                                                                     |
 | ---------------------------------------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
-| `new RedisQueue({ client, prefix? })`    | connected `redis@^4/^5` client with `watch()` + `multi()` | `prefix` — inherited from `MailTime` when omitted. Redis Cluster prefixes must map to one hash slot.        |
+| `new RedisQueue({ client, prefix?, useHashTags? })` | connected `redis@^4/^5` standalone or Cluster client | `prefix` — inherited from `MailTime` when omitted. Set `useHashTags: true` on Redis/KeyDB Cluster; set same option on JoSk Redis adapter. |
 | `new MongoQueue({ db, prefix? })`        | `db` from `MongoClient#db()`                         | `prefix` — inherited from `MailTime` when omitted. Indexes auto-created on first `ready()`.                  |
 | `new PostgresQueue({ client, prefix? })` | `pg.Pool` (recommended) or `pg.Client`               | `prefix` — inherited from `MailTime` when omitted. `mail_time_queue` table auto-migrated on first `ready()`. |
 
 For custom adapters see [docs/queue-api.md](https://github.com/veliovgroup/mail-time/blob/master/docs/queue-api.md).
 
+### Redis Cluster
+
+Set `useHashTags: true` on both `RedisQueue` and its JoSk Redis adapter. Queue state then uses `mailtime:{prefix}:letters` plus a tagged sorted-set schedule, so node-redis `createCluster()` can route every Lua operation to one slot. One prefix is one slot; shard high-volume traffic across prefixes. Tagged `iterate` returns at most 100 due rows per tick. Existing standalone keys are not read in this mode. Follow [Redis Cluster migration](https://github.com/veliovgroup/mail-time/blob/master/docs/migration-v5-v5.1.md) before cutover.
+
 ### Module functions
 
 - `mailTimePreset(name, overrides?)` → fresh MailTime constructor config. Deep-clones the named [preset](#settings-presets) and deep-merges your overrides (scalars win, nested `josk` composes). Throws on unknown `name` or non-object `overrides`.
@@ -502,6 +556,7 @@ For custom adapters see [docs/queue-api.md](https://github.com/veliovgroup/mail-
 ### Static
 
 - `MailTime.Template` — get/set the default HTML envelope template.
+- `MailTime.transportFrom(transport)` → `string | undefined`. Resolves a transport's sender address across `options.from`, `transporter.options.from`, `_defaults.from`, and `_options.from`, accepting both `'a@b.c'` and `{ name, address }`. Returns the first *usable* address; `undefined` when none resolves. Same value the `from(transport, details)` callback receives as `details.from`.
 
 ## Migrations
 
@@ -511,6 +566,8 @@ Upgrade checklists, adapter contract changes, and rollout notes live in the docs
 |------|-----|------------|
 | 3.x  | 4.1 | [v3 → v4](https://github.com/veliovgroup/mail-time/blob/master/docs/migration-v3-v4.md) |
 | 4.0  | 4.1 | [v4.0 → v4.1](https://github.com/veliovgroup/mail-time/blob/master/docs/migration-v4-v4.1.md) |
+| 4.1  | 5.0 | [v4.1 → v5](https://github.com/veliovgroup/mail-time/blob/master/docs/migration-v4.1-v5.md) |
+| 5.0  | 5.1 | [v5 → v5.1 Redis Cluster](https://github.com/veliovgroup/mail-time/blob/master/docs/migration-v5-v5.1.md) |
 
 New in v4 (opt-in): `mailTimePreset`, `concurrency`/`mode`, `sendingTimeout`, `drain()`/`pause()`/`resume()`, per-recipient handling, AI skills bundle.
 

package/adapters/redis.d.ts

@@ -4,7 +4,17 @@ export type RedisClient = {
     set: (key: string, value: string, options?: object) => Promise<unknown>;
     del: (key: string | string[]) => Promise<number>;
     ping: () => Promise<string>;
-    scanIterator: (options: object) => AsyncIterable<string | string[]>;
+    scanIterator?: ((options: object) => AsyncIterable<string | string[]>) | undefined;
+    hGet?: ((key: string, field: string) => Promise<string | null>) | undefined;
+    eval?: ((script: string, options: {
+        keys: string[];
+        arguments: string[];
+    }) => Promise<unknown>) | undefined;
+    evalSha?: ((sha: string, options: {
+        keys: string[];
+        arguments: string[];
+    }) => Promise<unknown>) | undefined;
+    scriptLoad?: ((script: string) => Promise<string>) | undefined;
     watch?: ((key: string) => Promise<unknown>) | undefined;
     unwatch?: (() => Promise<unknown>) | undefined;
     multi?: (() => object) | undefined;
@@ -12,6 +22,10 @@ export type RedisClient = {
 export type RedisQueueOption = {
     client: RedisClient;
     prefix?: string | undefined;
+    /**
+     * - Use Redis Cluster hash-tag keys (`mailtime:{prefix}:*`). Default keeps existing standalone keys.
+     */
+    useHashTags?: boolean | undefined;
 };
 /** Class representing Redis Queue for MailTime */
 export class RedisQueue {
@@ -22,8 +36,12 @@ export class RedisQueue {
     constructor(opts: RedisQueueOption);
     name: string;
     client: RedisClient;
+    useHashTags: boolean;
     prefix: any;
     uniqueName: string | undefined;
+    lettersKey: string | undefined;
+    scheduleKey: string | undefined;
+    concatKeysKey: string | undefined;
     /**
      * @async
      * @memberOf RedisQueue