@bumail/queue 0.12.0 → 0.14.0

package/docs/guide.md

@@ -7,6 +7,7 @@ workers share it.
 - [The smallest queue](#the-smallest-queue)
 - [Enqueuing](#enqueuing)
 - [Delivery and routing](#delivery-and-routing)
+- [MTA-STS](#mta-sts)
 - [Retries](#retries)
 - [Delivery status notifications](#delivery-status-notifications)
 - [Several workers, leases and stopping](#several-workers-leases-and-stopping)
@@ -164,7 +165,7 @@ outcome of the first session. A plain `5xx` (`5.1.x` unknown mailbox,
 the MX (an `SmtpError` coded `TLS_*`, a refused STARTTLS `454 4.7.0` under
 `mxTls: 'required'` included) never use it. A greylisting (a `4.7.x` whose
 text says grey/greylist) is the MX's to retry, and does not either. Only
-domains whose route is `'mx'` can fall back.
+domains whose route is `'mx'` can fall back. Under an `enforce` MTA-STS policy (`mtaSts`) nothing falls back, whatever failed: when every MX refuses the connection (4.4.1, 4.4.2) the recipient is deferred for the next attempt, since the smarthost would send the mail without the policy's TLS and name checks. In `testing` mode, or with no policy, it falls back as above. That holds for a reply too: under `enforce` no failure falls back, even a `5.7.x` reputation refusal from an MX that met the policy.
 
 A connection lost or a timeout *after* the final dot of `DATA` does not
 fall back either, though it reads as `4.4.2`: the MX may already have taken
@@ -260,6 +261,39 @@ createQueue({
 
 No resolver is needed when no route is `'mx'`.
 
+### Reusing connections
+
+By default the queue sends through a pool of connections it owns: a
+session to a server the queue already has an idle connection to (same
+host, address, port, EHLO name, TLS and credentials) sends its message
+over it, after an `RSET`, instead of opening a connection, a TLS
+handshake and an `AUTH` again. A connection carries 100 messages and
+waits 5 seconds idle, then closes. A `delivered`, `deferred` or `failed`
+event carries `reused: true` when the recipient's last session went over a
+connection that had carried a message before, `false` when it opened one.
+
+```ts
+createQueue({
+	store,
+	hostname: 'mail.example.net',
+	resolver,
+	reuse: { maxMessages: 50, idleSeconds: 10 }, // or the defaults, or `false`
+});
+```
+
+`reuse: false` opens one connection per session, as `sendMail` does. A
+`send` of your own is called as given: the queue never wraps it in a pool,
+and `reuse` does nothing with it, though `createQueue` still checks it.
+`reused` is on the events only if your `send` returns it.
+
+`stop()` closes the pool once the deliveries under way end, so no socket
+keeps the process alive; `start()` after it makes a new one. A connection
+that went stale while idle (the server closed it) is never a reason to
+fall back to a smarthost: the pool opens a new one to the same server.
+`@bumail/server` builds its own pool for its `send` and closes it after
+`queue.stop()`. The rules of [Falling back](#falling-back-to-a-smarthost) are
+the same with or without the pool.
+
 ### Timeouts
 
 `timeouts` and `deadline` pass to `sendMail`: each step's timeout in
@@ -316,6 +350,65 @@ A store keeps an item from being claimed before its message can be read
 at a claim is always damage, never an add still under way. Every store
 here adds both in one step.
 
+## MTA-STS
+
+`mtaSts` turns on MTA-STS (RFC 8461) for the recipient domains that go by MX. It needs `@bumail/mta-sts` (a peer) and a resolver with `txt`: `mtaSts.resolver`, or the queue's own `resolver` when it has `txt` (`@bumail/dns`'s does).
+
+```ts
+const REPORT_ENVID = 'my-report-7f3a'; // what you enqueue your own reports with: `{ from, to, envid: REPORT_ENVID }`
+const hosted = new Set(['example.net']); // domains you deliver in-process
+
+createQueue({
+	store,
+	hostname: 'mail.example.net',
+	resolver: nodeResolver(),
+	mtaSts: {
+		cache: createMemoryPolicyCache(), // the default; a PolicyCache of yours survives restarts
+		fetch: globalThis.fetch, // the default
+		fetchTimeoutMs: 60_000, // the whole policy fetch
+		maxAgeCap: 31_557_600, // seconds
+		skip: (item, domain) => item.envid === REPORT_ENVID || hosted.has(domain), // exempt from MTA-STS
+	},
+});
+```
+
+**Finding the policy.** The queue reads the TXT record `_mta-sts.<domain>`. A record whose `id` is the cached policy's, with the cache not expired, uses the cache. A new `id`, or an expired cache, fetches the policy and caches it until its `max_age` (on the queue's `clock`). No record and nothing cached means no policy. A fetch that fails (or a record that cannot be read) leaves a cached policy in force while it has not expired, as RFC 8461 §5.1 asks, and else no policy: the message is sent as without MTA-STS, and a `tls` event with `tlsResult: 'sts-policy-fetch-error'` (or `sts-policy-invalid`, `sts-webpki-invalid`) is told. A policy in mode `none` is no policy. The whole discovery (the cache, the TXT lookup and the fetch together) is bounded by one `fetchTimeoutMs`, not each step by its own (a lookup that times out is no record, a cache that times out, given at most a quarter of it, is empty). A policy fetch that fails is remembered for 5 minutes, per domain and record `id`, in the queue's memory: the sessions in between wait no longer for it, and each still tells the `sts-policy-fetch-error`. A new `id`, or a restart, fetches again. `stop()` aborts a lookup or fetch under way; the group is then left for later like one not started: nothing is sent, no attempt is counted, and the item stays pending. A cache entry that is malformed is ignored, and whatever else breaks the lookup is "no policy" and an `sts-policy-fetch-error` event, never a failed delivery.
+
+**Modes.**
+
+| mode | the session | on a failure |
+| --- | --- | --- |
+| `enforce` | `tls: 'required'` (over `mxTls`), `mxNames` = the policy's `mx` | an error with a `tlsResult` (an MX outside the policy: `MX_NOT_ALLOWED`; a certificate or STARTTLS failure) defers as `4.7.5`; never the `fallback` smarthost |
+| `testing` | as without a policy: your `mxTls`, every MX host | none; each try is told on `tls`, with what `enforce` would have refused (a name outside the policy, an unchecked certificate, no TLS) |
+| `none`, or no policy | as without MTA-STS | none |
+
+**Exemptions.** `skip(item, domain)` returns `true` for a session that must not be under MTA-STS: no discovery, no enforcement and no `tls` event. Use it for mail that must go despite a TLS failure and must not be counted (a mailed TLS-RPT report, RFC 8460 §5.3) and for domains you deliver in-process; mark the first with the `envid` of `enqueue`, which the item keeps. A predicate that throws exempts nothing.
+
+**Bouncing.** An `enforce` failure defers with the enhanced status `4.7.5` and the usual back-off. An attempt that may run past `retry.giveUpAfter` (it starts within `deadline` plus `fetchTimeoutMs` of it, which bounds the discovery) fetches the policy again, whatever the cache holds; whether the recipient fails as `5.7.5` or defers as `4.7.5` is decided when the failure happens, by the clock. A session in which any try broke the policy is a refusal even if its last error did not (a refused MX, then a connection failure). With that fresh policy: a policy that is gone (or in mode `none`) delivers the message normally; one still in `enforce` that fails bounces the recipient as `5.7.5` (a `failed` event and a DSN) instead of `4.4.7`. A cached policy that has not expired stays in force even when the TXT record has been removed; a domain that leaves MTA-STS publishes mode `none` first.
+
+**The `tls` event** has one payload per try at an MX address, and one for a policy that could not be fetched (no `mxHost`):
+
+```ts
+queue.on('tls', (event) => {
+	event.kind; // 'try' | 'policy' (a policy that could not be fetched)
+	event.at; // ms since the epoch, on the queue's clock
+	event.domain; // the recipient domain
+	event.policy; // { type: 'sts', domain, mode: 'enforce' | 'testing', mx, text } | { type: 'no-policy-found' }
+	event.mxHost; // absent for a fetch failure
+	event.address; // absent for a fetch failure and for an MX host the policy refused without a try
+	event.sendingAddress; // the local address of the connection, when the sender says
+	event.reused; // on the try that delivered: a connection the pool had used before
+	event.ok; // TLS held up; under a policy the MX name and the certificate matched
+	event.tlsResult; // RFC 8460 result type, absent when ok
+});
+```
+
+A try that failed before any TLS (the connection itself) is not told. In `enforce`, an MX host the policy skipped (or all of them, when none matches) is told once, without `address`, with `validation-failure`. Tries are told once the send ends.
+
+**Your own `send`.** The queue builds the MX options and gives them to `send`; with `mtaSts` they hold `mxNames`, `tls`, `onAttempt` and `onSkip` on top. A `send` that hands them to `sendMail` (or `createSmtpPool().sendMail`) is under the policy; one that drops them is not, and the queue cannot tell.
+
+**Not covered.** DANE, TLS reports (they are built from `tls` events by `@bumail/mta-sts`'s `tlsReport`), and the smarthost routes.
+
 ## Retries
 
 ```ts
@@ -543,6 +636,7 @@ off(); // stops listening
 | `delivered` | `{ id, from, recipient, reply?, attempts, route?, fallback?, mxReply? }`: `route` is `'mx'` or `'smarthost'`, `fallback` is `true` after the [fallback](#falling-back-to-a-smarthost), `mxReply` the MX's reply that led to it |
 | `deferred` | the same, and `nextAttemptAt`, and `mayHaveReachedMx: true` when the MX may hold a copy, `mayHaveReachedFallback: true` when the fallback smarthost may ([why](#falling-back-to-a-smarthost)) |
 | `failed` | the same as `delivered`, and `mayHaveReachedMx` and `mayHaveReachedFallback` as for `deferred` |
+| `tls` | `{ domain, policy, mxHost?, address?, ok, tlsResult? }`: one per try at an MX address under `mtaSts`, or per policy fetch failure; see [MTA-STS](#mta-sts) |
 | `dsn` | `{ kind: 'delayed' \| 'failed' \| 'relayed', id, of, to, recipients }`: `id` is the DSN's own item, `of` the item it reports on |
 | `error` | `{ error, id? }`: a store that failed (a lease renewal included), a DSN that could not be enqueued, a lease lost (`LEASE_LOST`), a message the store lost (`MESSAGE_UNREADABLE`), a route `sendMail` refused (`INVALID_OPTION`) |
 
@@ -1139,6 +1233,7 @@ connection or remove a file. It may be async.
 | `hostname` | `string` | — | EHLO, Reporting-MTA, the DSN's From |
 | `route` | `'mx' \| Smarthost` | `'mx'` | the default route |
 | `routes` | `Record<string, Route>` | — | a route per recipient domain |
+| `mtaSts` | `{ cache?, fetch?, resolver?, fetchTimeoutMs?, maxAgeCap?, skip? }` | off | MTA-STS on the MX route; see [MTA-STS](#mta-sts) |
 | `fallback` | `Smarthost` | — | where a recipient goes at once when MX fails by connection, timeout or a reputation refusal; see [Falling back](#falling-back-to-a-smarthost) |
 | `resolver` | `MxResolver` | — | needed by `'mx'` |
 | `mxPort` | `number`, 1–65535 | 25 | the port of MX hosts |
@@ -1159,5 +1254,6 @@ connection or remove a file. It may be async.
 | `pollInterval` | `number`, to 2147483647 | 5000 | ms between passes of `start()` |
 | `owner` | `string` | a random UUID | this worker's name on its leases |
 | `clock` | `{ now(): number }` | `Date.now` | the time |
-| `send` | `Sender` | `sendMail` | delivers one session; a custom one should return `dsnOffered` (leave it out and no `relayed` DSN is ever sent) |
+| `reuse` | `false \| { maxMessages?, idleSeconds? }` | `{ maxMessages: 100, idleSeconds: 5 }` | connection pool the queue owns and `stop()` closes; `false`: one connection per session; see [Reusing connections](#reusing-connections) |
+| `send` | `Sender` | the queue's pool, or `sendMail` with `reuse: false` | delivers one session; a custom one should return `dsnOffered` (leave it out and no `relayed` DSN is ever sent) |
 | `random` | `() => number` | `Math.random` | the jitter |

package/docs/roadmap.md

@@ -6,18 +6,17 @@ only number on it.
 
 ## Now
 
-Nothing under way.
+- **MTA-STS (RFC 8461) and TLS-RPT (RFC 8460)** ([#292](https://github.com/softistx/bumail/issues/292)) — a recipient
+  domain's policy fetched with `@bumail/mta-sts`, cached and enforced on MX
+  delivery: the certificate is checked against the policy's MX names. Opt-in
+  in the library, on in `@bumail/server`. An enforce failure defers the
+  message (`4.7.5`) and bounces it only at `giveUpAfter`. TLS reports are
+  recorded and sent opt-in.
 
 ## Next
 
-- **MTA-STS (RFC 8461) and TLS-RPT (RFC 8460)** — a recipient domain's
-  policy fetched, cached and enforced, `mxTls: 'required'` with the
-  certificate checked against the policy's MX names, and the reports.
 - **DANE (RFC 7672)** — TLSA records checked for MX hosts, once a
   resolver that validates DNSSEC can be given to the queue.
-- **One session for several messages** — the queue hands several items
-  for one domain to one SMTP session, once `@bumail/smtp`'s client reuses
-  connections.
 
 ## Later
 
@@ -46,6 +45,15 @@ Nothing under way.
 
 ## Shipped
 
+### 0.13.0
+
+- **One session for several messages** — the queue delivers several
+  messages for one MX over one connection, using `@bumail/smtp`'s pool of
+  reusable connections. On by default (100 messages per connection, idle 5
+  seconds); `reuse: false` turns it off, and a `send` of your own is never
+  wrapped. `stop()` closes the pool, and the events may carry `reused`. The
+  `QueueStore` contract does not change.
+
 ### 0.12.0
 
 - **The contract's specs, exported** — `@bumail/queue/conformance` exports

package/docs/troubleshooting.md

@@ -28,6 +28,9 @@ parts shown as … vary.
 - [`QueueError: timeouts.… must be a number of seconds above 0 and at most 2147483, not …`](#queueerror-timeouts-must-be-a-number-of-seconds-above-0-and-at-most-2147483-not-)
 - [`QueueError: store must be a QueueStore, such as MemoryQueueStore`](#queueerror-store-must-be-a-queuestore-such-as-memoryqueuestore)
 - [`QueueError: … must be an integer of at least …, not …`](#queueerror--must-be-an-integer-of-at-least--not-)
+- [`QueueError: reuse must be false or an object of limits, not …`](#queueerror-reuse-must-be-false-or-an-object-of-limits-not-)
+- [`QueueError: reuse.idleSeconds must be a number of seconds above 0 and at most 2147483, not …`](#queueerror-reuseidleseconds-must-be-a-number-of-seconds-above-0-and-at-most-2147483-not-)
+- [`QueueError: reuse.maxMessages must be an integer of at least 1, not …`](#queueerror-reusemaxmessages-must-be-an-integer-of-at-least-1-not-)
 - [`QueueError: dsn.returnContent must be 'headers' or 'full', not …`](#queueerror-dsnreturncontent-must-be-headers-or-full-not-)
 - [`QueueError: createQueue() takes an options object`](#queueerror-createqueue-takes-an-options-object)
 - [`QueueError: owner must be a non-empty string`](#queueerror-owner-must-be-a-non-empty-string)
@@ -96,6 +99,12 @@ parts shown as … vary.
 
 **Delivery**
 
+- [A recipient is deferred with `4.7.5`, or fails with `5.7.5`, `MTA-STS policy of … (enforce)`](#a-recipient-is-deferred-with-475-or-fails-with-575-mta-sts-policy-of--enforce)
+- [`MTA-STS discovery took more than N ms`](#mta-sts-discovery-took-more-than-n-ms-a-tls-event-with-tlsresult-sts-policy-fetch-error)
+- [`QueueError: mtaSts must be an object of options…`, `mtaSts.cache must be a PolicyCache with get and set`, `mtaSts.fetch must be a function`](#queueerror-mtasts-must-be-an-object-of-options-mtastscache-must-be-a-policycache-with-get-and-set-mtastsfetch-must-be-a-function)
+- [`QueueError: mtaSts.fetchTimeoutMs must be a number of milliseconds…`, `mtaSts.maxAgeCap must be a whole number of seconds…`, `mtaSts.skip must be a function`](#queueerror-mtastsfetchtimeoutms-must-be-a-number-of-milliseconds-mtastsmaxagecap-must-be-a-whole-number-of-seconds-mtastsskip-must-be-a-function)
+- [`QueueError: mtaSts needs a resolver with txt(name)`](#queueerror-mtasts-needs-a-resolver-with-txtname)
+- [A `tls` event with `sts-policy-fetch-error` and the mail goes out unprotected](#a-tls-event-with-sts-policy-fetch-error-and-the-mail-goes-out-unprotected)
 - [A recipient stays `deferred` with `4.4.1` or `4.4.2`](#a-recipient-stays-deferred-with-441-or-442)
 - [Every delivery fails at once with `5.7.1`, or with a reply naming your IP or EHLO name](#every-delivery-fails-at-once-with-571-or-with-a-reply-naming-your-ip-or-ehlo-name)
 - [A recipient fails with `5.1.3`, `The address is not one SMTP can carry`](#a-recipient-fails-with-513-the-address-is-not-one-smtp-can-carry)
@@ -302,6 +311,29 @@ or `… must be a number from … to …, not …`. A number option out of its r
 1), `dsn.delayAfter`, `limits.*` (`limits.maxReplyText` from 64 to
 900). Times are milliseconds.
 
+### `QueueError: reuse must be false or an object of limits, not …`
+
+**Code:** `INVALID`. `reuse` was `true`, `null` or another value. Give
+`false` for one connection per session, an object of limits, or leave it out
+for the defaults.
+
+```ts
+createQueue({ store, hostname, resolver, reuse: { maxMessages: 50 } });
+createQueue({ store, hostname, resolver, reuse: false });
+```
+
+### `QueueError: reuse.idleSeconds must be a number of seconds above 0 and at most 2147483, not …`
+
+**Code:** `INVALID`. `reuse.idleSeconds` is in seconds, not milliseconds:
+`5`, not `5000`. `reuse.maxMessages` has its own entry below.
+
+### `QueueError: reuse.maxMessages must be an integer of at least 1, not …`
+
+**Code:** `INVALID`. `reuse.maxMessages` is `0`, negative, fractional or not a
+number. The pool owns the limits and checks them when `createQueue` runs;
+the message is the pool's, prefixed with `reuse.`. Pass an integer of at
+least 1, or leave it out for the default (100).
+
 ### `QueueError: dsn.returnContent must be 'headers' or 'full', not …`
 
 **Code:** `INVALID`.
@@ -870,6 +902,49 @@ const clean = text.replaceAll('\0', '').toWellFormed();
 
 ## Delivery
 
+### A recipient is deferred with `4.7.5`, or fails with `5.7.5`, `MTA-STS policy of … (enforce)`
+
+**When:** `mtaSts` is set, the recipient domain publishes a policy in mode `enforce`, and the `deferred` event's `reply.status` is `4.7.5` (or the `failed` one's `5.7.5`) with a text such as `MTA-STS policy of example.com (enforce): No mail host of example.com (mx.other.net) is allowed by the policy's MX names (mail.example.com)`.
+**Why:** a session broke the policy: no MX host of the domain matches the policy's `mx` list (`MX_NOT_ALLOWED`), the MX does not offer STARTTLS, or its certificate does not check out for the MX name. The message is not sent anywhere else, `fallback` included (and under an `enforce` policy a plain connection failure of every MX is deferred too, never sent through the `fallback`). It is deferred with the usual back-off and bounced as `5.7.5` only at `retry.giveUpAfter`, after the policy is fetched once more.
+**Fix:** the `tls` events name the host and the `tlsResult` (`validation-failure` for a name outside the policy, `certificate-host-mismatch`, `certificate-expired`, `certificate-not-trusted`, `starttls-not-supported`). It is the receiving domain's to fix; if its policy is wrong, the message is delivered once the policy is. To send meanwhile, `retryNow(id)` after the domain changed its policy to `testing` or `none` and its record `id`.
+
+### `MTA-STS discovery took more than N ms` (a `tls` event with `tlsResult: 'sts-policy-fetch-error'`)
+
+**When:** `mtaSts` is set and the cache, the `_mta-sts` lookup and the policy fetch together used the whole `fetchTimeoutMs`.
+**Why:** the discovery has one deadline, not one per step, so a slow resolver or cache leaves the fetch less time. The message is sent without the policy (a cached one that has not expired stays in force). A policy fetch that failed is remembered for 5 minutes per domain and record `id`: a remote policy fixed under the same `id` is not fetched again for up to 5 minutes, and each session in between tells the same event. A timeout caused only by a budget the earlier steps had already used is not remembered.
+**Fix:** raise `fetchTimeoutMs`, or fix the slow resolver or cache; the domain's owner publishes a new `id` to have the policy fetched at once, or waits the 5 minutes.
+
+### `QueueError: mtaSts must be an object of options…`, `mtaSts.cache must be a PolicyCache with get and set`, `mtaSts.fetch must be a function`
+
+**Code:** `INVALID`. `mtaSts` is the options object, `cache` an object with `get` and `set` (`createMemoryPolicyCache()` fits), `fetch` a function.
+
+```ts
+mtaSts: { cache: createMemoryPolicyCache(), fetch: globalThis.fetch }
+```
+
+### `QueueError: mtaSts.fetchTimeoutMs must be a number of milliseconds…`, `mtaSts.maxAgeCap must be a whole number of seconds…`, `mtaSts.skip must be a function`
+
+**Code:** `INVALID`. `fetchTimeoutMs` is above 0 and at most 2147483647; `maxAgeCap` is a whole number of seconds from 0 to 31557600 (`MAX_AGE_LIMIT` of `@bumail/mta-sts`); `skip` is a function.
+
+```ts
+mtaSts: { fetchTimeoutMs: 30_000, maxAgeCap: 7 * 86_400 }
+```
+
+### `QueueError: mtaSts needs a resolver with txt(name)`
+
+**Code:** `INVALID`. `mtaSts` reads the `_mta-sts` TXT record, and neither `mtaSts.resolver` nor the queue's `resolver` has `txt`.
+
+```ts
+createQueue({ store, hostname, resolver: nodeResolver(), mtaSts: {} }); // nodeResolver() has txt
+createQueue({ store, hostname, resolver, mtaSts: { resolver: nodeResolver() } });
+```
+
+### A `tls` event with `sts-policy-fetch-error` and the mail goes out unprotected
+
+**When:** `mtaSts` is set, and the `tls` event has no `mxHost` and a `tlsResult` of `sts-policy-fetch-error`, `sts-policy-invalid` or `sts-webpki-invalid`.
+**Why:** the domain has an `_mta-sts` record, but `https://mta-sts.<domain>/.well-known/mta-sts.txt` could not be had (a timeout, a redirect, a status other than 200, a wrong content type, an invalid policy, a certificate the system does not trust). RFC 8461 §5.1 then applies a cached policy that has not expired, and with none sends without a policy: that session is not protected.
+**Fix:** the domain's to fix. Check the policy URL yourself, and that the `fetch` you gave (`mtaSts.fetch`) can reach it.
+
 ### A recipient stays `deferred` with `4.4.1` or `4.4.2`
 
 **When:** the `deferred` event's `reply` has no `code`, a `status` of

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@bumail/queue",
-	"version": "0.12.0",
+	"version": "0.14.0",
 	"description": "The outbound mail queue of a mail server: every recipient's state, retries with back-off, delivery status notifications (RFC 3464), leases for several workers, and a memory, a bun:sqlite, a PostgreSQL (Bun.sql) and a Redis (Bun.redis) store, delivering through @bumail/smtp/client",
 	"license": "MIT",
 	"type": "module",
@@ -73,12 +73,14 @@
 	"devDependencies": {
 		"@bumail/dns": "^0.2.2",
 		"@bumail/mime": "^0.2.0",
-		"@bumail/smtp": "^0.11.1",
+		"@bumail/mta-sts": "^0.1.0",
+		"@bumail/smtp": "^0.13.0",
 		"@types/bun": "^1.4.2"
 	},
 	"peerDependencies": {
 		"@bumail/mime": "^0.2.0",
-		"@bumail/smtp": "^0.11.1",
+		"@bumail/mta-sts": "^0.1.0",
+		"@bumail/smtp": "^0.13.0",
 		"typescript": "^6.0.3 || ^7.0.0"
 	},
 	"peerDependenciesMeta": {