cloudflare-email 0.1.0 → 0.3.0

data/templates/worker/README.md

@@ -1,37 +1,252 @@
 # cloudflare-email-ingress
 
 A Cloudflare Email Worker that forwards inbound mail to a Rails ActionMailbox
-ingress shipped with the [`cloudflare-email`](https://github.com/cole/cloudflare-email)
+ingress shipped with the [`cloudflare-email`](https://github.com/cole-robertson/cloudflare-email)
 gem.
 
 ## Deploy
 
+For a guided dashboard deployment that provisions resources and prompts for
+secrets, use the [Deploy to Cloudflare template](../deploy-to-cloudflare/README.md).
+It uses this same Worker with a standalone configuration. The CLI setup below
+is for the explicitly named development/staging/production environments.
+
 ```sh
-npm install
-wrangler secret put INGRESS_SECRET     # same value as cloudflare.ingress_secret in Rails credentials
-wrangler secret put RAILS_INGRESS_URL  # e.g. https://your-app.com/rails/action_mailbox/cloudflare/inbound_emails
-wrangler deploy
+npm ci
+npx wrangler r2 bucket create cloudflare-email-inbound-production
+npx wrangler queues create cloudflare-email-inbound-production
+npx wrangler secret put INGRESS_SECRET --env production
+npx wrangler secret put RAILS_INGRESS_URL --env production
+npm run deploy -- --env production
 ```
 
+Use Node 22 (at least 22.12), 24, or 26+. `INGRESS_SECRET` must match Rails' ingress secret;
+`RAILS_INGRESS_URL` is your public HTTPS endpoint, for example
+`https://your-app.com/rails/action_mailbox/cloudflare/inbound_emails`.
+The deploy and dev scripts require an explicit `development`, `staging`, or
+`production` environment. Each has its own Worker and secrets; set both secrets
+for each environment you use. For local development, run
+`npm run dev -- --env development`, with secrets in `.dev.vars.development`.
+Never put secrets into `wrangler.toml` or version control.
+
+The Worker refuses non-HTTPS destinations, URL credentials, and fragments.
+HTTP is allowed only for `localhost`, `127.0.0.1`, or `[::1]` local verification.
+The Worker and Rails ingress each default to a 25 MiB MIME size limit. Set
+`MAX_EMAIL_BYTES` to the same positive integer in both environments to change
+that limit. Both check actual body bytes; declared sizes alone are not trusted.
+Rails returns HTTP 413 for oversized mail and rejects malformed or stale signing
+headers before reading the body. Configure request size/rate limits and read
+timeouts at your reverse proxy as well: application limits do not bound earlier
+server buffering or the time spent receiving an HTTP request.
+
 Then in the Cloudflare dashboard:
 
 1. **Email Routing → Routes**
 2. Add a route for the address you want to receive on (e.g. `support@yourdomain.com`)
-3. Action: **Send to a Worker** → `cloudflare-email-ingress`
+3. Action: **Send to a Worker** → `cloudflare-email-ingress-production`
+
+Select the corresponding suffix when deploying another environment. Configure
+Email Routing DNS for the exact domain/subdomain first. Deploying this Worker
+does not change DNS or create routing rules.
 
 ## How it works
 
 For each inbound message, the Worker:
 
-1. Reads the raw RFC822 bytes from `message.raw`.
-2. Computes `HMAC-SHA256(INGRESS_SECRET, "{unix_timestamp}.{raw_body}")`.
-3. POSTs the raw bytes to `RAILS_INGRESS_URL` with:
-   - `Content-Type: message/rfc822`
-   - `X-CF-Email-Timestamp: <unix seconds>`
-   - `X-CF-Email-Signature: <hex digest>`
-4. If the Rails app responds non-2xx, the Worker calls `message.setReject` so
-   Cloudflare returns a delivery failure to the sender (the message is not
-   silently dropped).
+1. Validates and stores the exact RFC822 bytes and envelope in private R2.
+2. Queues a small pointer after storage succeeds.
+3. Signs and POSTs the stored message to Rails; only HTTP 2xx allows deleting it.
+4. Retries failures through the Queue and a minute-by-minute scheduled recovery
+   pass. Even exhausted queue retries leave the original message recoverable.
+
+The signed request uses HMAC-SHA256 and authenticates the original SMTP envelope
+alongside the MIME bytes. See [durable delivery setup and recovery](./docs/durable-inbound.md).
+`INBOUND_DELIVERY_MODE=direct` selects an explicit single-attempt fallback that
+calls `message.setReject` on non-2xx, timeout or network failure. Existing pending
+mail keeps draining through Queue and cron while this fallback is selected.
 
 The Rails controller verifies the signature in constant time and rejects
-timestamps older than 5 minutes (replay protection).
+timestamps outside its 5-minute acceptance window. A signature authenticates the
+Worker request, not the original email sender. Captured requests remain valid
+inside that window; Rails deduplicates messages through Action Mailbox.
+
+Deploy the Rails gem and this Worker together with ingress paused during the
+transition. Rails accepts v2 and v3 signatures and rejects v1/missing versions. It stores authenticated
+SMTP metadata separately from MIME; applications read it using
+`Cloudflare::Email::Envelope.for(inbound_email)`. Duplicate detection includes the
+exact SMTP recipient, preserving separate To/Cc/Bcc deliveries. The format accepts
+ASCII dot-atom addresses up to 254 bytes (local part up to 64 bytes), plus an empty
+sender for bounce messages. Other address forms are rejected before forwarding.
+
+## Reuse an existing Worker
+
+Keep your existing archiving, sender checks, and fallback policy. Copy this
+template's `src/index.js` into your Worker project and import either helper.
+`forwardEmail(message, env)` uses the same bounded reading, URL checks, timeout,
+and rejection policy as the explicit direct fallback. Optional metadata selects v3 signing:
+
+```js
+import { forwardEmail } from "./cloudflare-ingress.js";
+
+export default {
+  async email(message, env) {
+    // Supplied by your integration after validating its trusted provider context.
+    // This is your adapter function, not a Cloudflare EmailMessage API.
+    const providerData = await yourValidatedProviderContext(message, env);
+    return forwardEmail(message, env, {
+      metadata: { source: "your-provider", data: providerData },
+    });
+  },
+};
+```
+
+The bundled Worker does not collect provider authentication results. Never treat
+`Authentication-Results`, `Received-SPF`, or other sender-supplied MIME headers as
+trusted provider context. Signing metadata authenticates your Worker's assertion;
+it does not prove DMARC passed or authorize the original sender. Your application
+still owns sender policy, organization resolution, review, and document processing.
+
+If your Worker already owns transport and error handling, use the header builder:
+
+```js
+import { signedEmailHeaders } from "./cloudflare-ingress.js";
+
+const headers = await signedEmailHeaders({
+  secret: env.INGRESS_SECRET,
+  raw, // Original Uint8Array, obtained using your own bounded reader.
+  from: message.from,
+  to: message.to,
+  metadata: { source: "your-provider", data: providerData },
+});
+// Send these headers with exactly `raw`, using your existing transport.
+```
+
+The builder performs no network calls. You own destination validation, body size
+limits, timeouts, redirects, retries, and response handling. An optional `timestamp`
+in Unix seconds supports deterministic testing; production normally uses the
+default current time. Omitting `metadata` produces unchanged v2 headers. Invalid
+metadata throws from the builder; the forwarding helper rejects instead of
+silently falling back to v2.
+
+Metadata must contain exactly `source` and `data`. Source matches
+`[a-z][a-z0-9_.-]{0,127}`; data is a JSON object, with only JSON values and at most
+eight container levels including the outer metadata object. The UTF8 JSON is
+encoded as unpadded base64url, limited to 16,384 encoded characters, and sent as
+`X-CF-Email-Metadata`. Version 3 signs the exact bytes:
+
+```text
+HMAC-SHA256(secret, "v3.{timestamp}.{encoded_envelope}.{encoded_metadata}.{raw_body}")
+```
+
+Neither the MIME nor its headers are rewritten. Upgrade Rails to a version that
+supports v3 before enabling metadata in a custom Worker. Keep sensitive provider
+data out of metadata unless your application's retention and access policies
+permit storing it with email records.
+
+## Reuse the transport in an existing Worker
+
+`relayEmail` provides bounded reading, an optional archive, backend selection and
+one HTTP delivery attempt. Your Worker decides how a result maps to SMTP
+rejection, retry, fallback forwarding or logging. It does not call `setReject`
+or `forward` and does not extract or interpret sender authentication evidence.
+`forwardEmail` remains the explicit direct helper; the default export retains mail durably.
+
+```js
+import { relayEmail, archiveEmail, signedEmailHeaders } from "./cloudflare-ingress.js";
+
+export default {
+  async email(message, env) {
+    const outcome = await relayEmail(message, {
+      maxEmailBytes: 25 * 1024 * 1024,
+      timeoutMs: 15_000,
+      archiveTimeoutMs: 10_000,
+      // Omit archive when no bucket is configured. Key policy belongs to your app.
+      archive: env.EMAIL_ARCHIVE ? (args) => archiveEmail({
+        ...args, bucket: env.EMAIL_ARCHIVE,
+        key: `email/${crypto.randomUUID()}.eml`,
+      }) : undefined,
+      accepts: async ({ to }) => to.endsWith("@inbound.example.com"),
+      // Select only host-configured backends; never use a URL from an email header.
+      resolveBackend: async ({ from, to }) => ({
+        url: env.RAILS_INGRESS_URL, secret: env.INGRESS_SECRET,
+      }),
+      headers: ({ raw, from, to, backend }) => signedEmailHeaders({
+        raw, from, to, secret: backend.secret,
+      }),
+    });
+    // Example host policy: permanent rejection only for invalid/unaccepted mail;
+    // Throwing surfaces failure; do not assume Cloudflare retries SMTP delivery.
+    if (outcome.status === "rejected") message.setReject("email not accepted");
+    if (outcome.status === "failed") throw new Error("email relay unavailable");
+  },
+};
+```
+
+The relay reads the original stream once, enforcing the actual byte limit even
+when `rawSize` underreports it. It archives before checking `accepts` and before
+resolving the backend, so an archive can retain mail rejected by host policy.
+Invalid envelopes, unreadable streams and oversized messages are not archived.
+An archive exception or timeout does not prevent delivery: the result has
+`archiveFailed: true`. The archive deadline defaults to 10 seconds and can be
+changed with `archiveTimeoutMs`. It stops waiting but cannot cancel an R2 write:
+a timed-out archive may still finish later if the runtime remains alive. Monitor
+that flag if the archive is your outage recovery mechanism. Other callback
+implementations must finish within the Worker's runtime limits; the `timeoutMs`
+option applies to the HTTP request only.
+
+`accepts` must return exactly `true` to accept. `resolveBackend` returns an object
+with `url` and any additional host configuration needed by `headers`. Backend
+URLs must use HTTPS, or HTTP on localhost, `127.0.0.1` or `[::1]`; embedded
+credentials and fragments are rejected. Generic relay envelope checks enforce
+bounded strings without control characters, leaving address shape to the host.
+`signedEmailHeaders` retains the gem ingress's stricter ASCII address validation.
+
+The `headers` callback receives `{ raw, from, to, backend, archive }`, where
+`archive` is the archive callback's return value, or undefined on omission or
+failure. It can return a header object or `Headers`, including Basic credentials
+for an existing custom endpoint. The relay sends exactly `raw`, follows no
+redirects, cancels response bodies without reading them, and never includes
+backend response content or callback exception text in results. Callbacks are
+trusted host code: preserve the supplied bytes and keep secrets out of logs.
+Do not put an archive key or other per-attempt value into signed provider metadata;
+doing so changes the persisted message identity on retries.
+
+Every result has `status`, `reason` and `archiveFailed`. HTTP responses also add
+`httpStatus`:
+
+| Status | Reasons |
+| --- | --- |
+| `delivered` | `delivered` (HTTP 2xx) |
+| `rejected` | `invalid_envelope`, `too_large`, `not_accepted` |
+| `failed` | `unreadable`, `invalid_options`, `acceptance_failed`, `backend_failed`, `invalid_backend`, `headers_failed`, `timeout`, `fetch_failed`, `http_status` |
+
+All non-2xx responses, including redirects and permanent client errors, return
+`failed`/`http_status`; your host chooses which are permanent, retryable or eligible
+for fallback. The relay never retries on its own.
+
+`archiveEmail({ bucket, key, raw, from, to })` writes raw RFC822 bytes to an R2
+binding and returns `{ key }`. Keys must be nonempty, at most 1,024 UTF8 bytes and
+contain no ASCII control characters. It stores `message/rfc822` content type and
+ASCII-safe `from`, `to` and `size` custom metadata; non-ASCII envelope characters
+become `?` in this display metadata only. Retention, unique keys, archive browsing
+and recovery authorization belong to the host. Rails email persistence starts
+after delivery and does not replace this optional outage archive.
+
+## Durable inbound delivery
+
+[Durable inbound delivery](./docs/durable-inbound.md) is the default. Provision the
+configured private R2 bucket, Queue producer/consumer and cron before deploying.
+There is no rollout enable flag or automatic downgrade when bindings are missing.
+The guide covers upgrades, monitoring, recovery tests and explicit direct fallback.
+
+## Validate changes
+
+```sh
+npm ci
+npm test
+npm run check
+npm audit
+```
+
+Tooling dependencies are development-only; the deployed Worker imports no npm
+packages. `check` bundles a development deployment locally without publishing.

data/templates/worker/docs/durable-inbound.md

@@ -0,0 +1,149 @@
+# Durable inbound delivery
+
+When Rails is down, keep the email at Cloudflare and deliver it when Rails returns.
+The bundled Worker uses R2 storage, Queue delivery and scheduled recovery by
+default. No enable flag, Rails model, database, or tenancy change is needed.
+Missing infrastructure fails visibly; it never silently downgrades to direct delivery.
+
+Upgrade the Rails gem before deploying this Worker. New messages without a parseable
+Message-ID use a deterministic fallback for deduplication across Rails hosts.
+Older records created before this change use ActionMailbox's hostname-dependent
+fallback; replaying those historical messages after upgrading can create a new
+record. Reconcile any existing archive backlog before replaying it automatically.
+
+## Set it up
+
+Create a private R2 bucket and a Queue for each environment:
+
+```sh
+npx wrangler r2 bucket create cloudflare-email-inbound-production
+npx wrangler queues create cloudflare-email-inbound-production
+```
+
+The bundled `wrangler.toml` already configures these resources for each environment.
+Customize their names if needed. Configure **all three** pieces: the `INBOUND_EMAIL_STORE` R2
+binding, `INBOUND_EMAIL_QUEUE` producer and consumer, and the once-per-minute
+scheduled trigger. Keep `RAILS_INGRESS_URL` and `INGRESS_SECRET` configured as
+before. Deploy with `npm run deploy -- --env production`.
+
+Use separate buckets, queues and secrets for staging and production. Do not add
+an R2 lifecycle rule that expires `cloudflare-email/pending/` objects. Pending
+mail has no automatic expiration and can incur storage costs during a long
+outage. Do not publish the bucket or expose an unauthenticated recovery endpoint.
+Review storage access and retention as you would the Rails email database.
+Provision and deploy the infrastructure with Wrangler first; retain that configuration
+in your infrastructure repository. Subsequent `cloudflare:email:deploy_worker` Rails
+task uploads preserve R2/Queue bindings and check the bucket binding, queue producer
+binding, and minute schedule before changing code or secrets. This check does not
+validate the bucket lifecycle or queue consumer: verify those in your infrastructure
+configuration and live drill. The Ruby task does not provision resources.
+
+## Direct fallback and upgrades
+
+Set `INBOUND_DELIVERY_MODE = "direct"` in the environment's Wrangler vars only when
+you deliberately want a single HTTP handoff. With the Ruby task, pass
+`INBOUND_DELIVERY_MODE=direct bin/rails cloudflare:email:deploy_worker`.
+Direct mode permanently rejects on HTTP/network failure and cannot buffer an outage.
+Keep the existing R2/Queue bindings and cron during fallback: queue and scheduled
+recovery continue draining previously retained mail. To return to durable delivery,
+remove the Wrangler override or pass `INBOUND_DELIVERY_MODE=durable` to the Ruby task.
+Unknown values fail rather than choosing a transport implicitly.
+
+The old `DURABLE_INBOUND_ENABLED` rollout flag has been removed. Existing direct
+installations must provision the resources before upgrading the Worker, or explicitly
+select `INBOUND_DELIVERY_MODE=direct` before deploying. Upgrading the Ruby gem alone
+does not redeploy a Worker or change live routing.
+
+## What happens to a message
+
+1. The Worker validates the envelope and actual MIME size. Invalid or oversized
+   mail is permanently rejected, as in direct mode.
+2. It writes one R2 object containing the exact raw bytes, SMTP envelope, original
+   receive time, and optional provider metadata. It waits for the write before
+   returning from the Email Worker handler.
+3. It queues a small `{ version: 1, key }` pointer. If enqueueing fails, the stored
+   object remains discoverable by the scheduled sweep.
+4. The consumer signs the original bytes and metadata with a fresh timestamp and
+   POSTs to Rails. Only HTTP 2xx permits deleting the pending object; the queue
+   message is acknowledged only after that deletion completes.
+5. Timeout, connection failure, redirect, 401, 413 and all other non-2xx responses
+   retain the object and retry. Fix credentials or size configuration for errors
+   that cannot resolve themselves. There is no automatic discard on client errors.
+
+If Rails saved a message but its response was lost, replay sends the same message
+identity. Rails ingress deduplicates using the original bytes and envelope (and
+provider metadata for v3); changing the signing timestamp does not change that
+identity. A failed R2 deletion may also cause replay. A queue pointer to an
+already deleted object is a successful no-op and never resends mail. Do not reuse
+pending keys, alter saved context, or replace the ingress with an endpoint that
+acknowledges before durable ingestion. Retain Rails deduplication records for at
+least your maximum recovery horizon; manual restore/replay after Rails incineration
+can produce another message. Application processing must tolerate duplicates too.
+
+The scheduled handler scans five pending objects per minute and attempts their
+handoff directly. It persists an R2 pagination cursor and advances past failed
+objects, wrapping back to the beginning at the end. Poison messages therefore do
+not permanently block later pages. This sweep handles both enqueue failure and
+exhausted Queue retries without a separate dead-letter queue. During recovery,
+Queue consumers provide the primary throughput; scheduled-only recovery takes
+at least `ceil(pending_count / 5)` minutes per full pass. Overlapping cron runs
+may repeat a page; delivery remains idempotent. Scale and exercise this recovery
+rate against your expected backlog before depending on it.
+
+The template buffers MIME to sign it, and the durable path also builds a framed
+storage payload. Test messages near your configured size limit with realistic
+concurrent traffic against the Worker's memory limit before production rollout.
+Lower `MAX_EMAIL_BYTES` in both Worker and Rails if your workload requires it;
+lowering it below already retained message sizes leaves those messages pending
+until the limit is restored or they are recovered through an authorized process.
+
+## Operate and verify
+
+Enable Worker logs and alerts for the structured
+`component: "cloudflare_email_inbound"` events. They include a random storage key
+and fixed reason, never raw messages, addresses, secrets or HTTP response bodies.
+Alert on `storage_failed`, `enqueue_failed_retained`, repeated `queue_retry_retained`
+or `sweep_retry_retained`, Worker exceptions, and growth/age of objects under the
+pending prefix. `receivedAt` is inside the private stored frame; R2 object upload
+time is also available to infrastructure monitoring. Queue backlog alone is not
+sufficient: exhausted pointers may disappear while retained mail remains in R2.
+This template emits logs; configure your monitoring service to page the operator.
+
+In staging, stop Rails, send mail, confirm pending R2 objects, restore Rails, then
+verify one inbound message and removal of its pending object. Also verify recovery
+with the queue consumer disabled or retries exhausted, a lost HTTP response after
+Rails saved the message, invalid ingress credentials, and a failed R2 write.
+The repository includes unit coverage and a local Worker/Rails integration harness.
+Before changing ingress URLs, remember retained messages use the current configured
+URL and secret. Drain existing retained mail before repointing a bucket to another
+application or disabling the scheduled recovery trigger.
+
+For a custom Worker, import `retainEmail(message, env, { metadata })`,
+`consumeRetainedEmails(batch, env)` and `sweepRetainedEmails(env)` from `src/index.js`.
+Wire the latter two to your queue and scheduled handlers as in the default export.
+For a secondary raw archive, pass `archive: async ({ raw, from, to, key }) => ...`
+to `retainEmail`. It runs after the primary R2 commit with the original MIME bytes;
+`key` is the pending object identity. Its errors or default 10-second timeout are
+logged as `archive_failed_retained` and do not prevent queue delivery. A timed-out
+callback is not canceled. Set `archiveTimeoutMs` to an integer from 1 to 120,000
+milliseconds to fit your host's runtime budget. Treat bytes as read-only and never parse the private
+pending frame to build an archive. The primary R2 write is always mandatory.
+Trusted provider metadata is captured once at receipt, never recomputed on replay.
+The existing `relayEmail` archive callback is best effort and does not enable this
+durable protocol automatically.
+
+## Guarantee boundary
+
+After a successful R2 write, Rails outages and finite Queue retention do not
+discard the stored payload. This is at-least-once handoff, with Rails deduplication;
+it is not an absolute guarantee against all loss. R2 write failures, upstream
+SMTP acceptance behavior, account suspension, deleted infrastructure, misconfigured
+lifecycle rules and manual intervention remain outside that promise. A failed
+initial durable write throws and emits an error; it must not be mistaken for
+successful storage. Monitor that failure separately.
+
+Cloudflare documents [`setReject`](https://developers.cloudflare.com/email-service/api/route-emails/email-handler/)
+as a permanent SMTP rejection. Throwing an Email Worker error is not treated here
+as a documented durable SMTP retry contract. Cloudflare Queues has a
+[128 KB message limit and finite retention/retries](https://developers.cloudflare.com/queues/platform/limits/),
+which is why the queue stores only pointers and R2 retains the message independently.