mikser-io-post-email 2.0.0 → 11.1.0

package/.github/workflows/publish.yml

@@ -0,0 +1,73 @@
+# Publish to npm without a stored token.
+#
+# npm is retiring tokens that skip the second factor — its own token page says
+# "Publish new versions directly (deprecated — ends January 2027)". This is the
+# replacement: the job proves who it is at publish time with a short-lived
+# credential GitHub issues and npm verifies, so npm trusts "the publish.yml
+# workflow in this repository" rather than a string.
+#
+# There is no NODE_AUTH_TOKEN and no secret to configure. Nothing to leak,
+# expire or rotate — the two ways publishing broke in September 2026.
+#
+# Link it once on npm: the package → Settings → Trusted Publisher → GitHub
+# Actions, naming this repository and this filename. Until that exists the
+# publish step fails rather than falling back to something weaker.
+
+name: Publish
+
+on:
+  push:
+    tags: ['v*']
+  workflow_dispatch:
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      # The whole mechanism. Without it npm has nothing to verify.
+      id-token: write
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '24'
+          registry-url: 'https://registry.npmjs.org'
+
+      # Trusted publishing landed in npm 11.5.1.
+      - name: Use an npm that can do trusted publishing
+        run: npm install -g npm@latest
+
+      # `npm install`, not `npm ci`: these packages are developed in a
+      # workspace, so dependencies hoist to its root and each package's own
+      # lockfile is never read locally. They drift silently, and CI is the only
+      # thing that would ever read them.
+      - run: npm install --no-audit --no-fund
+        env:
+          # puppeteer downloads a browser on install; nothing in the release
+          # path renders a page.
+          PUPPETEER_SKIP_DOWNLOAD: '1'
+
+      # The tag is the release. A tag that disagrees with package.json would
+      # publish a version nobody asked for.
+      - name: Tag must match package.json
+        if: startsWith(github.ref, 'refs/tags/v')
+        run: |
+          TAG="${GITHUB_REF_NAME#v}"
+          PKG="$(node -p 'require("./package.json").version')"
+          if [ "$TAG" != "$PKG" ]; then
+            echo "tag v$TAG does not match package.json $PKG"
+            exit 1
+          fi
+          echo "publishing $PKG"
+
+      # The suite runs here now. It did not before: these packages declared
+      # mikser-io only as a peerDependency (npm does not install a package's
+      # own peers) or as file:../mikser-io (a path that exists on a laptop and
+      # nowhere else), so a standalone clone could not resolve the engine it
+      # tests against. Both are now real devDependencies with version ranges,
+      # which npm still satisfies from the workspace locally.
+      - run: npm test
+
+      - run: npm publish --access public

package/README.md

@@ -18,7 +18,8 @@ npm install mikser-io-post-email
 
 ```js
 // mikser.config.js
-import { documents, layouts, renderHbs, frontMatter } from 'mikser-io'
+import { documents, renderHbs, frontMatter } from 'mikser-io'
+import { layouts } from 'mikser-io-layouts'
 import { postMjml } from 'mikser-io-post-mjml'
 import { postEmail } from 'mikser-io-post-email'
 
@@ -45,11 +46,24 @@ export default {
 ---
 to: alice@acme.com
 subject: Welcome, Alice
-layout: welcome.html-mjml-email
+layout: welcome          # the layout's NAME — the chain suffixes are not part of it
 ---
 ```
 
-That's the transactional case: one entity, one recipient. The `.eml` lands at `out/welcome.eml` and the message ships immediately. On the next build, mikser's render manifest sees unchanged inputs and skips the whole chain — no resend.
+That's the transactional case: one entity, one recipient. The `.eml` lands at `out/welcome.eml` and the message is queued for immediate delivery — see [Delivery is out of band](#delivery-is-out-of-band). On the next build, mikser's render manifest sees unchanged inputs and skips the whole chain — no resend.
+
+## Delivery is out of band
+
+**Nothing is ever sent from inside the render pipeline.** A postprocessor writes the `.eml`, records a queue row, and returns; the transport is called later, by the drain.
+
+That matters because a transport is a third-party service. Sending inline — which this plugin did through 11.0.x — made two things true:
+
+- **Every cycle waited on the provider.** A build could not finish until every message in it had been acknowledged. One restart with a backlog spent it on 1515 sequential Mailgun round-trips, inside the pipeline, while the site's own requests queued behind it.
+- **The provider could fail the build.** A rejection — a 429, an outage, a socket that never answered — threw out of `postprocess` and failed the entity. A provider having a bad minute became a broken render, and with no marker written and no backoff, the next cycle simply tried again.
+
+Queuing costs one sqlite row and buys delivery that survives a crash mid-cycle, retries with backoff, a bounded per-message timeout, and a render that never waits on mail.
+
+Promptness is unaffected at both ends. A one-shot `mikser` drains at `onFinalized`, so it still delivers before it exits. A resident instance (`--watch` or `--server`) drains on a 60-second timer, off the cycle — and *not* at `onFinalized`, because that hook is awaited inside the cycle and draining there would put the transport straight back on the critical path.
 
 ## Recipient lists — `@listname` references
 
@@ -114,9 +128,9 @@ Resolution against `maxDelay` (default `'1h'`):
 
 | `sendAt` | What happens |
 |---|---|
-| missing / `'now'` | Deliver immediately during the chain |
+| missing / `'now'` | Write `.eml`, queue a row due now, deliver on the next drain |
 | future | Write `.eml`, queue a row, deliver on drain when due |
-| recent past (≤ `maxDelay`) | Catch-up: deliver immediately, warn nothing |
+| recent past (≤ `maxDelay`) | Catch-up: same as `'now'` |
 | ancient past (> `maxDelay`) | Write `.eml`, mark `expired_at`, log warning, no delivery |
 
 `maxDelay` is overridable per-entity:
@@ -135,16 +149,19 @@ A persistent table (`mikser_post_email_queue` in `runtime/mikser.sqlite`) holds
 ```sql
 mikser_post_email_queue (
     id          PRIMARY KEY → mikser_entities(id) ON DELETE CASCADE,
-    eml_path,
-    send_at, sent_at, expired_at,
-    attempts, last_error
+    eml_path,   eml_hash,   payload,
+    send_at,    next_attempt_at,
+    sent_at,    expired_at,
+    attempts,   last_error
 )
 ```
 
 - **PK on entity id + UPSERT** — re-editing `sendAt` reschedules in place; can't double-queue.
 - **FK CASCADE** — delete the source doc, queue row vanishes. Schedule follows the file.
-- **Drain triggers**: on startup, after every cycle (`onFinalized`), and every 60s in `--watch` mode.
-- **Failures stay queued**: `attempts` and `last_error` track retries; the next drain re-attempts.
+- **`payload` is what gets delivered** — the message as fields, stored on the row. Not the `.eml`: that is an audit artifact, and re-reading it to send a raw message depended both on the output tree still holding the file (a `--clear`, or a deploy `rsync --delete`, removes it) and on the transport honouring nodemailer's `raw` field. Mailgun's does not — `nodemailer-mailgun-transport` applies a key whitelist with no `raw` in it, so the field is dropped and what reaches the API has no sender, no recipient and no body. Released once the row is delivered.
+- **Drain triggers**: on startup; every 60s while resident (`--watch` **or** `--server`); after every cycle *only* when there is no timer, i.e. for a one-shot build.
+- **One drain at a time.** Passes are serialised. A pass only marks a row sent once the transport has answered, so an overlapping pass would select the same unmarked rows and deliver them twice — and a backlog easily outlasts the 60s interval.
+- **Failures back off**: `attempts` and `last_error` record what happened, and `next_attempt_at` holds the row until 1m × 2^attempts (capped at 15m). `send_at` is never moved — `maxDelay` is measured from it, so backing off must not make a row that keeps failing look permanently on-time and never expire.
 - **Retention**: delivered + expired rows are kept for `retention` (default `'90d'`) then pruned.
 
 ## Transport
@@ -233,6 +250,7 @@ Delete the entity's marker file, or bump `revision` to invalidate all of them.
 | `bcc` | spec | none | Global BCC, deduped with entity `bcc` |
 | `transport` | nodemailer config | JSON transport | Delivery target |
 | `maxDelay` | duration string | `'1h'` | How late past `sendAt` is still acceptable |
+| `sendTimeout` | duration string \| number | `'30s'` | How long one message may wait on the transport. `0` waits forever |
 | `retention` | duration string | `'90d'` | How long delivered/expired rows stay in the queue |
 | `sentFolder` | string | `'emails'` | Where send-once markers live; keep it gitignored and out of the deploy's delete set |
 | `revision` | number | `1` | Bump to invalidate every marker (forces a resend) |
@@ -242,8 +260,10 @@ Delete the entity's marker file, or bump `revision` to invalidate all of them.
 
 - **No transport-native scheduling.** Setting `send_at` on SendGrid/Mailchimp/etc. is not exposed — when an entity is rescheduled (frontmatter edited), the transport would hold both the old and new send. The internal queue dedupes via PK; transport-native scheduling can't.
 - **No rename-preserving queue.** A file rename = old entity DELETE + new entity INSERT; the old queue row cascades out, the new one's queue row inherits the renamed file's `sendAt`. Acceptable for v1.
-- **No throttling.** If 10k pending sends drain at once after long downtime, that's transport-side load to manage via SMTP-pool config.
-- **No retry policy beyond "next drain tries again".** `attempts` is recorded for visibility; there's no exponential backoff or dead-lettering.
+- **No throttling.** A drain delivers every due row, one at a time, with no rate cap. After long downtime that is transport-side load to manage via SMTP-pool config. Deliberately not capped per pass: a cap plus `maxDelay` is a way to *expire* mail that was only late because we throttled it, and a lost email is worse than a slow one.
+- **No dead-lettering.** A row that keeps failing backs off to a 15m retry and is eventually expired by `maxDelay`, with `last_error` on the row. There is no separate dead-letter table and no alert.
+- **A timeout cannot cancel a send.** It abandons the call, so a message that timed out may still have been delivered and the retry can duplicate it. The trade is deliberate: an unbounded send wedges the (serialised) drain permanently, which stops *all* delivery.
+- **One transport per process.** `transport` and the drain timer are module-level, so configuring `postEmail()` twice in one project has the second call's transport win for both. Use one.
 
 ## License
 

package/index.js

@@ -19,6 +19,8 @@ import {
     markerName,
     formatMarker,
     isDelivered,
+    sendWithTimeout,
+    backoffDelay,
 } from './lib/pure.js'
 
 // Re-export pure helpers so callers (and tests) can import them
@@ -28,6 +30,8 @@ export {
     humanizeMs,
     resolveAddresses,
     decideTiming,
+    sendWithTimeout,
+    backoffDelay,
 } from './lib/pure.js'
 
 // Postprocessor name — used in chain syntax (`welcome.html-mjml-email.hbs`)
@@ -39,23 +43,44 @@ export const output = 'eml'
 // prepend `mikser_`. `mikser-io-post-email` → `mikser_post_email_*`.
 registerSchema('post_email', `
     CREATE TABLE IF NOT EXISTS mikser_post_email_queue (
-        id          TEXT PRIMARY KEY REFERENCES mikser_entities(id) ON DELETE CASCADE,
-        eml_path    TEXT NOT NULL,
-        eml_hash    TEXT,
-        send_at     INTEGER NOT NULL,
-        sent_at     INTEGER,
-        expired_at  INTEGER,
-        attempts    INTEGER NOT NULL DEFAULT 0,
-        last_error  TEXT
+        id              TEXT PRIMARY KEY REFERENCES mikser_entities(id) ON DELETE CASCADE,
+        eml_path        TEXT NOT NULL,
+        eml_hash        TEXT,
+        -- The message as fields (JSON), which is how it is DELIVERED.
+        --
+        -- Not the .eml on disk: that is an audit artifact, and re-reading it
+        -- to send a raw message made delivery depend both on the output tree
+        -- still holding the file (a --clear, or a deploy rsync with --delete,
+        -- removes it) and on the transport honouring nodemailer's raw field
+        -- at all. Mailgun's does not: nodemailer-mailgun-transport applies a
+        -- key whitelist with no raw in it, so the field is dropped and what
+        -- reaches the API has no sender, no recipient and no body.
+        --
+        -- Cleared on delivery; a sent row does not need to keep the body.
+        payload         TEXT,
+        -- When delivery was ASKED for. maxDelay is measured from it, so it is
+        -- never moved once written.
+        send_at         INTEGER NOT NULL,
+        -- When the next attempt may run. Set by a failed attempt's backoff;
+        -- separate from send_at precisely so backing off cannot make a row
+        -- that keeps failing look permanently on-time and never expire.
+        next_attempt_at INTEGER,
+        sent_at         INTEGER,
+        expired_at      INTEGER,
+        attempts        INTEGER NOT NULL DEFAULT 0,
+        last_error      TEXT
     );
     CREATE INDEX IF NOT EXISTS idx_mikser_post_email_queue_due
         ON mikser_post_email_queue (send_at)
         WHERE sent_at IS NULL AND expired_at IS NULL;
 `)
 
-const DEFAULT_MAX_DELAY_MS = 60 * 60 * 1000                 // 1h
-const DEFAULT_RETENTION_MS = 90 * 24 * 60 * 60 * 1000       // 90d
-const DRAIN_INTERVAL_MS    = 60 * 1000                      // 60s in watch mode
+const DEFAULT_MAX_DELAY_MS    = 60 * 60 * 1000              // 1h
+const DEFAULT_RETENTION_MS    = 90 * 24 * 60 * 60 * 1000    // 90d
+const DEFAULT_SEND_TIMEOUT_MS = 30 * 1000                   // per message
+const DRAIN_INTERVAL_MS       = 60 * 1000                   // while resident
+const BACKOFF_BASE_MS         = 60 * 1000                   // 1st retry
+const BACKOFF_MAX_MS          = 15 * 60 * 1000              // ceiling
 
 // Per-config closures populate this on onLoaded. Module-level so the
 // postprocess() call (which is per-entity, may run on workers in
@@ -64,6 +89,14 @@ const DRAIN_INTERVAL_MS    = 60 * 1000                      // 60s in watch mode
 let transport = null
 let drainTimer = null
 
+// The drain is SINGLE-FLIGHT, and has to be: a pass delivers sequentially and
+// only marks a row sent once the transport has answered, so a second pass
+// starting while the first is still in flight selects the same unmarked rows
+// and delivers them a second time. There are two callers (the timer, and
+// onFinalized for one-shot builds) and a backlog can easily outlast the 60s
+// interval — 1515 queued messages did, on gpoint.bg.
+let draining = false
+
 // ---------- EML composition + delivery -------------------------------
 
 // Build the .eml bytes nodemailer would have handed SMTP. Used both
@@ -137,19 +170,21 @@ async function recordSentSafely(config, id, hash, logger) {
 
 // ---------- queue ops ------------------------------------------------
 
-function upsertQueueRow({ id, emlPath, emlHash, sendAt }) {
+function upsertQueueRow({ id, emlPath, emlHash, sendAt, payload }) {
     useDatabase().handle.prepare(`
-        INSERT INTO mikser_post_email_queue (id, eml_path, eml_hash, send_at)
-        VALUES (?, ?, ?, ?)
+        INSERT INTO mikser_post_email_queue (id, eml_path, eml_hash, payload, send_at)
+        VALUES (?, ?, ?, ?, ?)
         ON CONFLICT(id) DO UPDATE SET
-            send_at    = excluded.send_at,
-            eml_path   = excluded.eml_path,
-            eml_hash   = excluded.eml_hash,
-            sent_at    = NULL,
-            expired_at = NULL,
-            attempts   = 0,
-            last_error = NULL
-    `).run(id, emlPath, emlHash, sendAt)
+            send_at         = excluded.send_at,
+            eml_path        = excluded.eml_path,
+            eml_hash        = excluded.eml_hash,
+            payload         = excluded.payload,
+            sent_at         = NULL,
+            expired_at      = NULL,
+            next_attempt_at = NULL,
+            attempts        = 0,
+            last_error      = NULL
+    `).run(id, emlPath, emlHash, payload == null ? null : JSON.stringify(payload), sendAt)
 }
 
 function recordExpiredInBand({ id, emlPath, sendAt, reason }) {
@@ -166,13 +201,82 @@ function recordExpiredInBand({ id, emlPath, sendAt, reason }) {
     `).run(id, emlPath, sendAt, Date.now(), reason)
 }
 
-function markSent(id)        { useDatabase().handle.prepare(`UPDATE mikser_post_email_queue SET sent_at = ? WHERE id = ?`).run(Date.now(), id) }
-function markExpired(id, r)  { useDatabase().handle.prepare(`UPDATE mikser_post_email_queue SET expired_at = ?, last_error = ? WHERE id = ?`).run(Date.now(), r, id) }
-function markFailed(id, err) { useDatabase().handle.prepare(`UPDATE mikser_post_email_queue SET attempts = attempts + 1, last_error = ? WHERE id = ?`).run(err.message || String(err), id) }
+// `payload` is released here: the row stays for its retention window as a
+// record that this id was delivered, and a delivered message body has no
+// further use — keeping every rendered email in the cache DB for 90 days does.
+function markSent(id)        { useDatabase().handle.prepare(`UPDATE mikser_post_email_queue SET sent_at = ?, payload = NULL WHERE id = ?`).run(Date.now(), id) }
+// payload released for the same reason as in markSent — an expired row is
+// never delivered, so holding its body for the retention window is pure cost.
+function markExpired(id, r)  { useDatabase().handle.prepare(`UPDATE mikser_post_email_queue SET expired_at = ?, last_error = ?, payload = NULL WHERE id = ?`).run(Date.now(), r, id) }
+// A failed attempt stays queued, but not due again immediately. Retrying a
+// flapping provider every 60s only multiplies the load on it, and for an
+// attempt that TIMED OUT rather than been refused it multiplies the
+// duplicates — the message may well have gone out. So back off exponentially
+// in `next_attempt_at` and leave `send_at` untouched: `send_at` is the
+// delivery intent and maxDelay is measured from it, so moving it would make a
+// row that keeps failing look permanently on-time and never expire.
+//
+// Returns the delay it set, for the caller to log.
+function markFailed(id, err) {
+    const row = useDatabase().handle
+        .prepare(`SELECT attempts FROM mikser_post_email_queue WHERE id = ?`).get(id)
+    const delay = backoffDelay({ attempts: row?.attempts, baseMs: BACKOFF_BASE_MS, maxMs: BACKOFF_MAX_MS })
+    useDatabase().handle.prepare(`
+        UPDATE mikser_post_email_queue
+        SET attempts = attempts + 1, last_error = ?, next_attempt_at = ?
+        WHERE id = ?
+    `).run(err.message || String(err), Date.now() + delay, id)
+    return delay
+}
+
+// Hand one message to the transport under a per-message timeout — see
+// sendWithTimeout in lib/pure.js for why an unbounded send is not an option.
+// `sendTimeout: 0` opts out.
+function sendMailWithTimeout({ config, payload }) {
+    return sendWithTimeout({
+        send: message => transport.sendMail(message),
+        payload,
+        timeoutMs: parseDuration(config.sendTimeout, DEFAULT_SEND_TIMEOUT_MS),
+    })
+}
+
+// What to hand the transport for a queued row: the stored fields, or — for a
+// row queued by a version that did not store them — the composed .eml re-read
+// from disk. That fallback is best-effort by nature: it survives only while
+// the file is still in the output tree, and only reaches the provider intact
+// on a transport that honours `raw`, which SMTP does and Mailgun's does not.
+async function deliveryPayload({ row, logger }) {
+    if (row.payload) return JSON.parse(row.payload)
+
+    logger.warn(
+        'postEmail: %s was queued before the message payload was stored — falling back to its .eml. ' +
+        'A transport that ignores `raw` (Mailgun) will reject it; re-render the entity to requeue it properly.',
+        row.id)
+
+    const emlAbs = path.isAbsolute(row.eml_path)
+        ? row.eml_path
+        : path.join(runtime.options.outputFolder, row.eml_path)
+    return { raw: await readFile(emlAbs) }
+}
 
 // Drain the queue: deliver due rows that are still within maxDelay,
 // expire the overdue ones. Failed deliveries stay queued for retry.
+//
+// Serialised — see `draining`. Overlapping passes double-send.
 async function drain({ config, logger }) {
+    if (draining) {
+        logger.debug('postEmail: a drain is already in flight, skipping this pass')
+        return
+    }
+    draining = true
+    try {
+        await drainQueue({ config, logger })
+    } finally {
+        draining = false
+    }
+}
+
+async function drainQueue({ config, logger }) {
     const db = useDatabase()
     if (!db?.isOpen) return
 
@@ -188,10 +292,11 @@ async function drain({ config, logger }) {
     `).run(now - retentionMs, now - retentionMs)
 
     const due = db.handle.prepare(`
-        SELECT id, eml_path, eml_hash, send_at FROM mikser_post_email_queue
+        SELECT id, eml_path, eml_hash, payload, send_at FROM mikser_post_email_queue
         WHERE sent_at IS NULL AND expired_at IS NULL AND send_at <= ?
+          AND (next_attempt_at IS NULL OR next_attempt_at <= ?)
         ORDER BY send_at
-    `).all(now)
+    `).all(now, now)
 
     for (const row of due) {
         // Cascade-race guard: catalog delete may have fired between
@@ -226,15 +331,12 @@ async function drain({ config, logger }) {
         }
 
         try {
-            const emlAbs = path.isAbsolute(row.eml_path)
-                ? row.eml_path
-                : path.join(runtime.options.outputFolder, row.eml_path)
-            const raw = await readFile(emlAbs)
+            const payload = await deliveryPayload({ row, logger })
             if (config.dryRun) {
                 logger.info('postEmail: [dryRun] would deliver %s', row.id)
                 markSent(row.id)
             } else {
-                await transport.sendMail({ raw })
+                await sendMailWithTimeout({ config, payload })
                 // Order matters: the mail is out, so retire the row FIRST.
                 // If the marker write were inside this try and threw, the
                 // catch below would markFailed() and leave the row due —
@@ -245,8 +347,9 @@ async function drain({ config, logger }) {
             }
             logger.info('postEmail: delivered %s', row.id)
         } catch (err) {
-            markFailed(row.id, err)
-            logger.error('postEmail: delivery failed for %s — %s', row.id, err.message || err)
+            const delay = markFailed(row.id, err)
+            logger.error('postEmail: delivery failed for %s, retrying in %s — %s',
+                row.id, humanizeMs(delay), err.message || err)
         }
     }
 }
@@ -303,28 +406,48 @@ export async function postprocess({ entity, options, config, logger }) {
     const maxDelayMs = parseDuration(entity.meta?.maxDelay ?? config.maxDelay, DEFAULT_MAX_DELAY_MS)
     const timing = decideTiming({ meta: entity.meta ?? {}, maxDelayMs })
 
-    if (timing.mode === 'queue') {
-        upsertQueueRow({ id: entity.id, emlPath: entity.destination, emlHash: hash, sendAt: timing.sendAt })
-        logger.info('postEmail: queued %s for %s', entity.id, new Date(timing.sendAt).toISOString())
-    } else if (timing.mode === 'expired') {
+    if (timing.mode === 'expired') {
         const reason = `overdue by ${humanizeMs(timing.overdueMs)}, past maxDelay ${humanizeMs(maxDelayMs)}`
         recordExpiredInBand({ id: entity.id, emlPath: entity.destination, sendAt: timing.sendAt, reason })
         logger.warn('postEmail: %s expired in-band — %s', entity.id, reason)
+        return { success: true, result: entity.destination }
+    }
+
+    // Both 'now' and 'queue' go through the queue. NOTHING is delivered from
+    // here any more.
+    //
+    // `postprocess` runs INSIDE the render pipeline. 'now' used to await
+    // transport.sendMail() right here, which made two things true that should
+    // never have been:
+    //
+    //   1. Every cycle waited on a third-party service. A build could not
+    //      finish until the provider had answered for every message in it —
+    //      1515 sequential Mailgun round-trips, in one case, inside the
+    //      pipeline, while the site's own requests queued behind it.
+    //   2. The provider could FAIL THE BUILD. A rejection — a 429, an
+    //      outage, a socket that never answered — threw out of postprocess
+    //      and failed the entity, so a provider having a bad minute became a
+    //      broken render, with no retry: the marker was never written, so the
+    //      next cycle simply tried again with no backoff.
+    //
+    // A row costs one sqlite insert and buys what the inline path never had:
+    // delivery that survives a crash mid-cycle, retries with backoff, a
+    // bounded per-message timeout, and a render that never waits on mail.
+    //
+    // Delivery promptness is preserved at both ends. A one-shot build drains
+    // at onFinalized, so `mikser` still sends before it exits. A resident
+    // instance drains on the timer, off the cycle, within DRAIN_INTERVAL_MS.
+    upsertQueueRow({
+        id: entity.id,
+        emlPath: entity.destination,
+        emlHash: hash,
+        payload: { from, to, cc, bcc, subject, html },
+        sendAt: timing.sendAt,
+    })
+    if (timing.mode === 'now') {
+        logger.info('postEmail: queued %s for immediate delivery', entity.id)
     } else {
-        // mode === 'now' — deliver synchronously, once.
-        try {
-            if (config.dryRun) {
-                logger.info('postEmail: [dryRun] would deliver %s', entity.id)
-            } else {
-                await transport.sendMail({ from, to, cc, bcc, subject, html })
-                // Outside the throw path on purpose — see recordSentSafely.
-                await recordSentSafely(config, entity.id, hash, logger)
-            }
-            logger.info('postEmail: delivered %s', entity.id)
-        } catch (err) {
-            logger.error('postEmail: delivery failed for %s — %s', entity.id, err.message || err)
-            throw err
-        }
+        logger.info('postEmail: queued %s for %s', entity.id, new Date(timing.sendAt).toISOString())
     }
 
     return { success: true, result: entity.destination }
@@ -339,32 +462,38 @@ export function postEmail(config = {}) {
         const logger = useLogger()
         transport = nodemailer.createTransport(config.transport ?? { jsonTransport: true })
 
-        // Migrate pre-1.1 installs: the queue table predates eml_hash, and
-        // CREATE TABLE IF NOT EXISTS won't add a column to an existing one.
+        // Bring an older queue table up to date. CREATE TABLE IF NOT EXISTS
+        // will not add a column to a table that already exists, so each one
+        // added since needs its own ALTER.
         //
         // Only "already there" is expected and silent. Anything else — a
-        // locked or read-only database — must be said out loud: swallowing
-        // it leaves the table without the column, and the failure resurfaces
-        // later as an opaque "no column named eml_hash" from an INSERT
-        // inside postprocess, far from its cause.
-        try {
-            const db = useDatabase()
-            if (db?.isOpen) db.handle.exec(`ALTER TABLE mikser_post_email_queue ADD COLUMN eml_hash TEXT`)
-        } catch (err) {
-            if (!/duplicate column/i.test(err.message || '')) {
-                logger.error('postEmail: could not add the eml_hash column — %s', err.message || err)
-            }
-        }
-
-        onFinalized(async () => {
+        // locked or read-only database — must be said out loud: swallowing it
+        // leaves the table without the column, and the failure resurfaces
+        // later as an opaque "no column named …" from an INSERT inside
+        // postprocess, far from its cause.
+        for (const column of ['eml_hash TEXT', 'payload TEXT', 'next_attempt_at INTEGER']) {
             try {
-                await drain({ config, logger })
+                const db = useDatabase()
+                if (db?.isOpen) db.handle.exec(`ALTER TABLE mikser_post_email_queue ADD COLUMN ${column}`)
             } catch (err) {
-                logger.error('postEmail: drain failed (onFinalized) — %s', err.message || err)
+                if (!/duplicate column/i.test(err.message || '')) {
+                    logger.error('postEmail: could not add the %s column — %s', column, err.message || err)
+                }
             }
-        })
+        }
 
-        if (runtime.options.watch && !drainTimer) {
+        // The timer is what makes delivery out-of-band, so it has to run
+        // whenever this process stays up — `--server` as much as `--watch`.
+        // Gated on `watch` alone it was missing from the configuration that
+        // needs it most: a production `--server` instance (no watcher, which
+        // is how gpoint-cms runs) had no timer at all, so a queued message
+        // waited for the end of the next cycle and, after the last cycle,
+        // waited indefinitely.
+        //
+        // Matches how core decides the same thing — see the residency check
+        // in mikser-io's src/instance.js.
+        const resident = runtime.options.watch || runtime.options.server
+        if (resident && !drainTimer) {
             drainTimer = setInterval(() => {
                 drain({ config, logger }).catch(err => {
                     logger.error('postEmail: drain failed (timer) — %s', err.message || err)
@@ -373,13 +502,41 @@ export function postEmail(config = {}) {
             drainTimer.unref?.()
         }
 
-        // Run one drain at startup to catch anything that came due
-        // while mikser was off. Wrapped so a startup-time delivery
-        // failure doesn't crash the boot.
-        try {
-            await drain({ config, logger })
-        } catch (err) {
-            logger.error('postEmail: startup drain failed — %s', err.message || err)
+        onFinalized(async () => {
+            // A one-shot build has no timer and exits after finalize, so the
+            // queue MUST be drained here or its mail is never sent.
+            //
+            // When a timer owns delivery, this must NOT drain: onFinalized is
+            // awaited inside the cycle, so draining here would put the
+            // third-party transport straight back on the critical path — the
+            // coupling the queue exists to break.
+            //
+            // Keyed on the timer actually existing rather than on residency,
+            // so the two conditions cannot disagree and strand the queue with
+            // neither draining it.
+            if (drainTimer) return
+            try {
+                await drain({ config, logger })
+            } catch (err) {
+                logger.error('postEmail: drain failed (onFinalized) — %s', err.message || err)
+            }
+        })
+
+        // One drain at startup, to catch whatever came due while mikser was
+        // off. Awaited only when nothing else will drain: a resident instance
+        // gets this fired and forgotten, because awaiting it here is how a
+        // restart with a backlog spent its boot inside the transport —
+        // 1515 messages, one at a time, before the first cycle ran.
+        if (drainTimer) {
+            drain({ config, logger }).catch(err => {
+                logger.error('postEmail: startup drain failed — %s', err.message || err)
+            })
+        } else {
+            try {
+                await drain({ config, logger })
+            } catch (err) {
+                logger.error('postEmail: startup drain failed — %s', err.message || err)
+            }
         }
     })
 
@@ -397,6 +554,6 @@ export function postEmail(config = {}) {
         name: config.name ?? 'email',
         output,
         options: config,
-        postprocess,
+        postprocess, module: import.meta.url,
     }
 }

package/lib/pure.js

@@ -92,6 +92,57 @@ export function decideTiming({ meta, maxDelayMs, now = Date.now() }) {
     return                        { mode: 'expired', sendAt: ts, overdueMs: now - ts }
 }
 
+// ---------- delivery containment -------------------------------------
+//
+// A transport is a THIRD-PARTY SERVICE — Mailgun's HTTP API, an SMTP relay,
+// SES — and none of its behaviour is ours to control. These two decide how
+// much of that is allowed to reach the rest of the process.
+
+// Hand one message to `send`, with a bound on how long that may take.
+//
+// nodemailer's own connection and socket timeouts cover only its SMTP
+// transport; an HTTP API transport can sit on a socket for as long as the OS
+// allows. Unbounded, one hung call stops ALL delivery: the drain is
+// single-flight, so that pass never finishes, no later pass starts, and
+// nothing is ever sent again — with nothing in the log to say why.
+//
+// A timeout ABANDONS the call; it cannot cancel it. The provider may still
+// deliver the message, so a retry after a timeout can produce a duplicate.
+// That is the deliberate trade — one possible duplicate against delivery
+// stopping permanently — and it is why the retry backs off rather than firing
+// again on the next pass. A falsy `timeoutMs` opts out and waits forever.
+export async function sendWithTimeout({ send, payload, timeoutMs }) {
+    if (!timeoutMs) return send(payload)
+
+    let timer
+    try {
+        return await Promise.race([
+            send(payload),
+            new Promise((_, reject) => {
+                timer = setTimeout(
+                    () => reject(new Error(`the transport did not answer within ${humanizeMs(timeoutMs)}`)),
+                    timeoutMs)
+            }),
+        ])
+    } finally {
+        // Always — a resolved send must not leave a pending timer holding the
+        // event loop open for the rest of the timeout.
+        clearTimeout(timer)
+    }
+}
+
+// How long to wait before attempt number `attempts + 1`.
+//
+// Retrying a flapping provider on every pass only multiplies the load on it,
+// and for an attempt that timed out rather than been refused it multiplies
+// the duplicates. Exponential, capped, and counted from attempts already
+// made, so the first failure waits `baseMs` and a persistent one settles at
+// `maxMs` instead of hammering.
+export function backoffDelay({ attempts, baseMs, maxMs }) {
+    const made = Math.max(0, Math.floor(Number(attempts) || 0))
+    return Math.min(baseMs * 2 ** made, maxMs)
+}
+
 // ---------- send-once ledger (pure parts) ----------------------------
 //
 // A durable on-disk marker lets postEmail skip re-sending an email it

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mikser-io-post-email",
-  "version": "2.0.0",
+  "version": "11.1.0",
   "description": "Email postprocessor for mikser-io — sends rendered output via SMTP and writes .eml audit files. Composes after post-mjml in a chain.",
   "main": "index.js",
   "type": "module",
@@ -18,9 +18,12 @@
   },
   "homepage": "https://github.com/almero-digital-marketing/mikser-io-post-email#readme",
   "peerDependencies": {
-    "mikser-io": "^10.0.0"
+    "mikser-io": "^11.0.1"
   },
   "dependencies": {
     "nodemailer": "^6.9.0"
+  },
+  "devDependencies": {
+    "mikser-io": "^11.0.1"
   }
 }

package/test/unit.test.js

@@ -8,6 +8,7 @@ import {
     resolveSpec, dedupe, resolveAddresses,
     decideTiming,
     deliveryHash, markerName, formatMarker, isDelivered, normalizeSendAt,
+    sendWithTimeout, backoffDelay,
 } from '../lib/pure.js'
 
 describe('parseDuration', () => {
@@ -298,3 +299,80 @@ describe('formatMarker / isDelivered', () => {
         assert.equal(isDelivered(undefined, 1, h), false)
     })
 })
+
+describe('sendWithTimeout', () => {
+    it('passes the payload through and returns what the transport returned', async () => {
+        const seen = []
+        const result = await sendWithTimeout({
+            send: async message => { seen.push(message); return { messageId: 'ok' } },
+            payload: { to: ['a@example.com'] },
+            timeoutMs: 1000,
+        })
+        assert.deepEqual(seen, [{ to: ['a@example.com'] }])
+        assert.deepEqual(result, { messageId: 'ok' })
+    })
+
+    it('rejects when the transport does not answer in time', async () => {
+        await assert.rejects(
+            sendWithTimeout({
+                send: () => new Promise(() => {}),   // never settles
+                payload: {},
+                timeoutMs: 20,
+            }),
+            /did not answer within/)
+    })
+
+    it('lets a real transport error through unchanged', async () => {
+        await assert.rejects(
+            sendWithTimeout({
+                send: async () => { throw new Error('421 rate limited') },
+                payload: {},
+                timeoutMs: 1000,
+            }),
+            /421 rate limited/)
+    })
+
+    it('waits forever when the timeout is falsy', async () => {
+        // Opting out must not wrap the call at all: a send that takes longer
+        // than any plausible timeout still resolves.
+        const result = await sendWithTimeout({
+            send: () => new Promise(resolve => setTimeout(() => resolve('late'), 30)),
+            payload: {},
+            timeoutMs: 0,
+        })
+        assert.equal(result, 'late')
+    })
+
+    it('does not leave a pending timer behind on success', async () => {
+        // A leaked timer keeps the event loop alive for the rest of the
+        // timeout — for a one-shot build that is the process refusing to exit.
+        const before = process.getActiveResourcesInfo().filter(r => r === 'Timeout').length
+        await sendWithTimeout({ send: async () => 'sent', payload: {}, timeoutMs: 60_000 })
+        const after = process.getActiveResourcesInfo().filter(r => r === 'Timeout').length
+        assert.equal(after, before)
+    })
+})
+
+describe('backoffDelay', () => {
+    const opts = { baseMs: 60_000, maxMs: 900_000 }
+
+    it('waits the base delay after the first failure', () => {
+        assert.equal(backoffDelay({ attempts: 0, ...opts }), 60_000)
+    })
+    it('doubles with each attempt already made', () => {
+        assert.equal(backoffDelay({ attempts: 1, ...opts }), 120_000)
+        assert.equal(backoffDelay({ attempts: 2, ...opts }), 240_000)
+        assert.equal(backoffDelay({ attempts: 3, ...opts }), 480_000)
+    })
+    it('caps at maxMs instead of growing without bound', () => {
+        assert.equal(backoffDelay({ attempts: 4, ...opts }), 900_000)
+        assert.equal(backoffDelay({ attempts: 40, ...opts }), 900_000)
+    })
+    it('treats a missing or junk attempt count as none made', () => {
+        // attempts comes straight from a sqlite row, which can be null on a
+        // row written before the column existed.
+        assert.equal(backoffDelay({ attempts: null, ...opts }), 60_000)
+        assert.equal(backoffDelay({ attempts: undefined, ...opts }), 60_000)
+        assert.equal(backoffDelay({ attempts: -3, ...opts }), 60_000)
+    })
+})