cloudflare-email 0.4.0 → 0.4.2

data/docs/reference.md

@@ -0,0 +1,190 @@
+# Ruby and Rails reference
+
+For a first setup, follow the [Rails quickstart](getting-started.md).
+
+## Plain Ruby
+
+```ruby
+require "cloudflare-email"
+
+client = Cloudflare::Email::Client.new(
+  account_id: ENV.fetch("CLOUDFLARE_ACCOUNT_ID"),
+  api_token: ENV.fetch("CLOUDFLARE_API_TOKEN"),
+)
+
+response = client.send(
+  from: { address: "hello@mail.example.com", name: "Example" },
+  to: "you@example.net",
+  subject: "Hello",
+  text: "Plain body",
+  html: "<p>HTML body</p>",
+  reply_to: "support@in.example.com",
+)
+response.message_id             # Provider ID when returned
+response.delivered              # Immediately delivered recipients
+response.queued                 # Recipients queued for later delivery
+response.permanent_bounces      # Recipients that permanently bounced
+response.suppressed_recipients  # Recipients dropped by suppression policy
+```
+
+`to` may be omitted for cc-only or bcc-only mail. Addresses accept strings or `{ address:, name: }` hashes. Attachments use Cloudflare's `content` (base64), `filename`, `type`, `disposition`, and optional `content_id` fields. For full MIME control, use `client.send_raw(from:, recipients:, mime_message:)`.
+
+An HTTP success does not mean every recipient received the message. Inspect the recipient outcome arrays and use [delivery events](delivery-events.md) for later results. 
+
+Cloudflare currently limits ordinary sends to 50 recipients and 5 MiB including attachments; verified destination addresses have a 25 MiB allowance. These limits are enforced by Cloudflare. See [limits](https://developers.cloudflare.com/email-service/platform/limits/) and the [header allowlist](https://developers.cloudflare.com/email-service/reference/headers/).
+
+## Receive through ActionMailbox
+
+Cloudflare Email Routing invokes an Email Worker. The bundled Worker forwards unchanged raw MIME to Rails with a versioned HMAC-SHA256 signature covering the timestamp, SMTP envelope, and body. Rails verifies the signature and a five-minute timestamp window before storing the message in ActionMailbox.
+
+```sh
+bin/rails generate cloudflare:email:install
+```
+
+The interactive installer offers ActionMailbox installation/migrations and a default `MainMailbox`, copies the Worker, and configures ingress in development and production. Add `--all-envs` to include test; configure custom staging environments explicitly. Send-only apps do not need ActionMailbox.
+
+Add the generated ingress secret and an optional management token:
+
+```yaml
+cloudflare:
+  account_id: your-account-id
+  api_token: your-runtime-send-token
+  management_token: your-deployment-token
+  ingress_secret: the-generated-random-secret
+```
+
+Equivalent environment names are `CLOUDFLARE_MANAGEMENT_TOKEN` and `CLOUDFLARE_INGRESS_SECRET`. Keep management credentials in the deployment environment rather than the running application. Credentials stored in the application's Rails credentials are accessible to that application; this gem is not a secret-isolation boundary.
+
+1. Add the receiving subdomain, for example `in.example.com`, in **Email Routing → apex domain → Settings → Subdomains**. This is a separate onboarding step from sending.
+2. Deploy the environment's Worker and create its route:
+
+First create the private R2 bucket and Queue, then deploy the generated Wrangler
+configuration using the [durable setup guide](../templates/worker/docs/durable-inbound.md).
+The Ruby task below updates that provisioned Worker while preserving its bindings.
+It refuses a missing bucket/queue binding or recovery schedule before changing code.
+
+```sh
+RAILS_ENV=production bin/rails cloudflare:email:deploy_worker URL=https://app.example.com/rails/action_mailbox/cloudflare/inbound_emails
+RAILS_ENV=production bin/rails cloudflare:email:provision_route ADDRESS=support@in.example.com
+```
+
+Subdomain provisioning checks configured DNS before creating a rule. It never enables the parent apex on behalf of a subdomain. A missing setup or permission fails with instructions. DNS records alone do not prove propagation or live routing; send a real test afterward.
+
+For a zone apex, provisioning may enable Email Routing and its DNS records. Only use this when Cloudflare should handle mail for that apex. `provision_catchall DOMAIN=example.com` changes the **zone-wide** catch-all; a subdomain that resolves to a parent zone is rejected. See [subdomain onboarding](https://developers.cloudflare.com/email-service/configuration/subdomains/).
+
+Replace the scaffolded mailbox's `process` with your application logic. For tenant or mailbox selection, use the authenticated SMTP recipient rather than the sender-controlled MIME `To`/`Cc` headers:
+
+```ruby
+class ApplicationMailbox < ActionMailbox::Base
+  routing ->(inbound) { Cloudflare::Email::Envelope.for(inbound)&.fetch("to")&.match?(/\Asupport@/i) } => :support
+end
+```
+
+`Cloudflare::Email::Envelope.for(inbound_email)` returns a string-keyed `{"from" => "sender@example.com", "to" => "support@example.com"}` hash, or `nil` when the record has no authenticated envelope (for example, another ingress). The metadata is stored on the raw-email blob before routing jobs enqueue. It does not modify the MIME source. Envelope sender information records the SMTP reverse path; it does not authenticate the human sender. An empty `from` is valid for bounces.
+
+The bundled Worker uses v2 signatures by default; missing or v1 signatures are rejected. The custom-ingress API also supports opt-in v3 signatures carrying authenticated Worker metadata. See [custom ingestion](custom-ingress.md) for signed Worker metadata. Both envelope versions require ASCII dot-atom addresses, at most 254 bytes with a 64-byte local part. Quoted local parts, address literals, and internationalized addresses are not supported by this envelope format.
+
+Successful ingress storage returns HTTP 200; duplicate storage returns 200 too. The timestamp window limits request age, but is not a one-time replay ledger. Version 2 deduplication includes the exact SMTP recipient, so identical MIME delivered to separate To/Cc/Bcc recipients creates separate inbound records while a retry for the same recipient creates none.
+
+The Worker has a 15-second Rails request timeout and rejects redirects. Its default [durable inbound path](../templates/worker/README.md#durable-inbound-delivery) saves messages in R2 before acceptance, then retries Rails handoffs through Queues and scheduled recovery. Provision the resources before deploying. `INBOUND_DELIVERY_MODE=direct` selects the single-attempt fallback, which can reject mail during an outage; existing retained mail keeps recovering. Rails storage acceptance does not guarantee later mailbox-job success. Monitor Rails jobs and Cloudflare Worker logs.
+
+### Local development and deployment
+
+For a simple development tunnel, use the explicit direct fallback. Start Rails, then:
+
+```sh
+INBOUND_DELIVERY_MODE=direct bin/rails cloudflare:email:deploy_worker
+bin/rails cloudflare:email:dev
+```
+
+The dev task requires `cloudflared`, refuses environments other than development, and updates the existing development Worker's URL to a temporary tunnel. It forces a dedicated origin Host; development middleware permits only POSTs to the email ingress on that Host. The task checks that the guard is running before opening the tunnel. Configure a separate development receiving address and route. Stopping the tunnel leaves that URL in the development Worker until the next update. A Worker deployed without a URL rejects mail until the tunnel sets it.
+
+For a custom installer `--worker-dir`, pass `SCRIPT=custom-directory/src/index.js` to the Ruby `deploy_worker` task. The installer prints the corresponding command.
+
+Ruby deployment and Wrangler use matching names: `cloudflare-email-ingress-development`, `-staging`, and `-production`. Initial durable infrastructure deployment uses Wrangler and requires Node 22.12+ (or a supported newer version):
+
+```sh
+cd cloudflare-worker
+npm ci
+npx wrangler secret put INGRESS_SECRET --env production
+npx wrangler secret put RAILS_INGRESS_URL --env production
+npm run deploy -- --env production
+```
+
+See the [Worker README](../templates/worker/README.md).
+
+Rotate the shared ingress secret in Rails and the corresponding Worker during a coordinated deployment. This version has no overlapping-key rotation window; requests can fail while secrets differ.
+
+## Retry and configuration
+
+`Client.new` options are also accepted by `config.action_mailer.cloudflare_settings`:
+
+| Option | Default |
+|---|---|
+| `account_id`, `api_token` | Required |
+| `base_url` | `https://api.cloudflare.com/client/v4` |
+| `timeout` | 30 seconds for open/read/write |
+| `total_timeout` | Defaults to `timeout`; bounds one complete HTTP attempt, including headers and streamed body |
+| `max_response_bytes` | 1 MiB for `Client`; 32 MiB for `EventConsumer` queue batches |
+| `retries` | 3 additional attempts |
+| `initial_backoff` | 0.5 seconds, doubling |
+| `max_retry_after` | 60 seconds |
+| `retry_ambiguous` | `false` |
+| `logger` | `nil`, optional `warn` logger |
+
+By default, only 429 responses and pre-send connection failures retry. Numeric and HTTP-date `Retry-After` values are honored up to the cap. Read/write timeouts, connection resets, and 5xx responses may occur after acceptance; they raise without automatically resending. Setting `retry_ambiguous: true` restores retries for those failures and can send duplicates.
+
+Responses are streamed with a byte limit, including decompressed content and
+error responses. A total deadline or response-size failure after transmission
+does not prove a send failed: the outbox preserves the ambiguous claim and blocks
+automatic resend. Net::HTTP's implicit retries are disabled. Explicit retries
+and backoff can make a logical call longer than `total_timeout`; queue handlers,
+database transactions and a complete polling run have separate runtime budgets.
+An uncertain queue acknowledgement permits receipt replay, never email resending.
+
+No idempotency key is sent. Reusing Message-ID does not guarantee deduplication or exactly-once delivery. Account for ActiveJob's retry policy too: retrying the whole mailer job can resend even when this client's retries are disabled.
+
+Errors inherit from `Cloudflare::Email::Error`: `ConfigurationError`, `AuthenticationError`, `ValidationError`, `RateLimitError`, `ServerError`, and `NetworkError`. API errors expose `status` and parsed `response`.
+
+## Observability and permissions
+
+Notifications: `cloudflare_email.send` / `send_raw` include `account_id`, `path`, `status`, `message_id`, and all four recipient outcome arrays. `cloudflare_email.ingress` includes `bytes`, `result` (`ok`, `duplicate`, `bad_signature`, `stale`, `too_large`), and the stored `message_id` when available. `cloudflare_email.delivery_event` wraps handler execution with `event_id`, `message_id`, and lifecycle `status`; it does not report queue acknowledgement completion. Instrumentation errors include ActiveSupport's exception metadata.
+
+| Task | Purpose / credentials |
+|---|---|
+| `doctor` | Read diagnostics with runtime token; limited read permissions are reported |
+| `send_test FROM=... TO=...` | Send a real message using runtime send permission |
+| `deploy_worker URL=https://...` | Management token: Workers Scripts Edit |
+| `provision_route ADDRESS=...` | Management token: Zone Read, Email Routing Rules Edit; DNS Read for subdomain checks; routing-settings write permission for apex enablement |
+| `provision_catchall DOMAIN=...` | Same routing management permissions; changes the zone-wide catch-all |
+| `dev` | Management token: Workers Scripts Edit; development only |
+| `consume_events` | Separate Queues Read/Write token, queue ID, configured handler |
+| `deliver OPERATION_KEY=...` | Dispatch a saved outbox operation using sending credentials |
+| `replay_events [MESSAGE_ID=...]` | Replay durable receipts for the configured account |
+| `pending_deliveries` | List prepared or uncertain outbox operations for operator review |
+
+The optional Rails adapter emits `cloudflare_email.outbox_prepare`,
+`cloudflare_email.outbox_send`, and `cloudflare_email.outbox_reconcile` notifications
+with operation identity and resulting state. These do not contain MIME or API
+tokens. A notification is method instrumentation; an enclosing application
+transaction may still roll back. Inspect the durable ledger for authoritative state.
+
+Management tasks fall back to the runtime token if `management_token` is unset. Restrict scopes and accounts to the operations you need. Event consumers use a separate `queues_token` and do not fall back to a send token.
+
+## SMTP alternative
+
+Cloudflare also supports authenticated SMTP. Existing Rails SMTP applications can use it without this gem's delivery method:
+
+```ruby
+config.action_mailer.delivery_method = :smtp
+config.action_mailer.smtp_settings = {
+  address: "smtp.mx.cloudflare.net",
+  port: 465,
+  ssl: true,
+  authentication: :plain,
+  user_name: "api_token",
+  password: ENV.fetch("CLOUDFLARE_SMTP_TOKEN"),
+}
+```
+
+Cloudflare documents Email Sending Edit permission for SMTP, implicit TLS on port 465, and no outbound STARTTLS on 587. See [SMTP documentation](https://developers.cloudflare.com/email-service/api/send-emails/smtp/). Remove the generated `:cloudflare` initializer override if switching to SMTP.

data/docs/routing-diagnostics.md

@@ -1,7 +1,5 @@
 # Check whether an address is configured to reach your Worker
 
-This diagnostic is available in gem version 0.3.0.
-
 A mailbox registered in Rails is not proof that Cloudflare can deliver to it.
 Use this read-only check when onboarding a receiving domain, investigating a
 missing message, or checking an existing catch-all before activating addresses.

data/docs/thread-correlation.md

@@ -16,10 +16,8 @@ message's ID. Keep unmatched replies available for application handling when an
 ID is missing or unknown. Send acceptance and saving the returned ID are not one
 atomic transaction.
 
-The old `SecureMessageId` helper was removed before the 0.2 release. Cloudflare
-controls Message-ID and our [live test](verification/2026-09-10-live.md) confirmed
-replacement of custom IDs for both raw MIME and ActionMailer sends. The provider's
-returned IDs matched received headers and supported reply correlation.
+Cloudflare controls the outgoing Message-ID. Store the returned provider ID
+instead of relying on a custom ID supplied before sending.
 
 Email headers locate a conversation; they do not authorize access to it. Scope
 lookups by the trusted SMTP recipient obtained through `Envelope.for(inbound)`.

data/docs/troubleshooting.md

@@ -9,7 +9,7 @@ into public logs or issues.
 
 | What you see | What to check next |
 | --- | --- |
-| New generator or API is missing | Check `Gemfile.lock`. Use cloudflare-email 0.4.x with mailbox-kit 0.1.x; follow the [installation guide](getting-started.md#install-the-current-code). |
+| New generator or API is missing | Check `Gemfile.lock`. Use cloudflare-email 0.4.x with mailbox-kit 0.1.x; follow the [installation guide](getting-started.md#1-install). |
 | Credentials appear to be ignored | Nonempty Rails credentials override environment variables. Check the Rails environment and restart the app after changes. |
 | Authentication or domain error | Run `bin/rails cloudflare:email:doctor`; check account ID, token permissions and sending-domain verification. |
 | `doctor` reports limited read access | A send token may lack diagnostic read permissions. Review the specific result; diagnostics alone cannot prove whether sending works. |
@@ -30,7 +30,7 @@ FROM=hello@mail.example.com TO=you@example.net bin/rails cloudflare:email:send_t
 | --- | --- |
 | Address receives nothing | Verify receiving-subdomain onboarding and the address route's Worker. Sending-domain verification is separate. |
 | Route provisioning refuses a subdomain | Complete Email Routing subdomain setup and DNS preflight first. Do not enable or replace apex routing just to bypass the error. |
-| Worker reports 401 from Rails | Check matching ingress secrets and compatible signature versions. The Worker defaults to v2. The opt-in v3 [custom-ingress API](custom-ingress.md) requires upgrading Rails before enabling Worker metadata. |
+| Worker reports 401 from Rails | Check matching ingress secrets and compatible signature versions. The Worker defaults to v2. Use the [custom-ingress API](custom-ingress.md) for v3 Worker metadata. |
 | Worker reports 408 | Check clock accuracy and the five-minute signing window. |
 | Worker reports 413 or rejects size | Raw MIME exceeds the configured limit. Match `MAX_EMAIL_BYTES` on Rails and Worker and check upstream request limits. |
 | Worker reports 3xx | Point it directly at the final HTTPS ingress URL; redirects are intentionally rejected. |
@@ -42,7 +42,7 @@ FROM=hello@mail.example.com TO=you@example.net bin/rails cloudflare:email:send_t
 
 ## Local development
 
-Restart Rails after upgrading so the ingress-only guard is installed. Start
+Restart Rails after configuring the gem so the ingress-only guard is installed. Start
 Rails on the expected port before `cloudflare:email:dev`, install `cloudflared`,
 and use development credentials and a development receiving route.
 
@@ -99,6 +99,6 @@ bin/rails cloudflare:email:replay_events
 ```
 
 See [delivery events](delivery-events.md) for queue setup and
-[observability](../README.md#observability-and-permissions) for notifications you
+[observability](reference.md#observability-and-permissions) for notifications you
 can connect to your monitoring. A delivery notification wraps handler execution;
 it is not proof that queue acknowledgement completed.

data/lib/cloudflare/email/client.rb

@@ -15,6 +15,7 @@ module Cloudflare
       DEFAULT_BACKOFF  = 0.5
       DEFAULT_MAX_RESPONSE_BYTES = 1_048_576
       MAX_RETRY_AFTER  = 60 # seconds; never sleep longer than this even if server says so
+      PROVIDER_TEXT_LIMIT = 200 # characters of an unstructured error body quoted in a message
 
       RETRYABLE_NETWORK = [
         Net::OpenTimeout, Net::ReadTimeout, Net::WriteTimeout, Errno::ECONNRESET,
@@ -159,7 +160,7 @@ module Cloudflare
           do_request(method, uri, body)
         rescue *RETRYABLE_NETWORK => e
           unless PRE_SEND_NETWORK.any? { |type| e.is_a?(type) } || @retry_ambiguous
-            raise NetworkError.new("provider response could not be read; delivery outcome is unknown; automatic retry disabled"), cause: nil
+            raise NetworkError.new("provider response could not be read; request outcome is unknown; automatic retry disabled"), cause: nil
           end
           raise NetworkError.new("provider request could not be completed; request outcome is unknown"), cause: nil if attempts > @retries
           log_retry(attempts, e)
@@ -235,7 +236,7 @@ module Cloudflare
 
       def handle_response(response, raw = response.body)
         status  = response.code.to_i
-        body    = parse_body(raw)
+        body    = parse_body(raw, status)
         retry_after = response["Retry-After"]
 
         # Stash Retry-After on the error response so retry logic can use it.
@@ -267,11 +268,28 @@ module Cloudflare
         end
       end
 
-      def parse_body(raw)
+      # A 2xx that is not readable JSON fails closed: the request may have taken
+      # effect. A non-2xx was refused, so a body that is not a Cloudflare JSON
+      # object (a plain-text throttle, a bare JSON string) is reported as the
+      # provider's own words rather than as an unknown outcome.
+      def parse_body(raw, status)
         return {} if raw.nil? || raw.empty?
-        JSON.parse(raw)
+        parsed = JSON.parse(raw)
+        return parsed if parsed.is_a?(Hash) || (200..299).cover?(status)
+        unstructured_error(status, parsed.is_a?(String) ? parsed : raw)
       rescue JSON::ParserError
-        { "errors" => [{ "message" => "invalid JSON API response; request outcome is unknown" }] }
+        return { "errors" => [{ "message" => "invalid JSON API response; request outcome is unknown" }] } if (200..299).cover?(status)
+        unstructured_error(status, raw)
+      end
+
+      def unstructured_error(status, text)
+        text = text.to_s.dup.force_encoding(Encoding::UTF_8).scrub("").gsub(/[[:cntrl:][:space:]]+/, " ").strip
+        # Markup is an error page, not a reason; never echo it.
+        text = "invalid JSON API response" if text.empty? || text.start_with?("<")
+        text = "#{text[0, PROVIDER_TEXT_LIMIT]}..." if text.length > PROVIDER_TEXT_LIMIT
+        # A 5xx may have failed after taking effect; a 4xx/429 was refused.
+        text = "#{text}; request outcome is unknown" if status >= 500
+        { "errors" => [{ "message" => "HTTP #{status}: #{text}" }] }
       end
 
       def extract_message(body)

data/lib/cloudflare/email/ingress.rb

@@ -44,9 +44,17 @@ module Cloudflare
         # missing membership without scheduling Rails processing again.
         def receive_into_mailbox!
           result = nil
-          Mailboxes.receive(recipient: envelope.fetch("to")) do
-            result = persist_rails_record
-            result.record
+          prepared = nil
+          recipient = envelope.fetch("to")
+          # Upload before the membership transaction opens (see InboundEmail).
+          Mailboxes.with_recipient(recipient: recipient) { prepared = prepare_rails_record }
+          begin
+            Mailboxes.receive(recipient: recipient) do
+              result = prepared.persist
+              result.record
+            end
+          ensure
+            Mailboxes.for_tenant(prepared.tenant_key) { prepared.discard } if prepared
           end
           result.record if result.created?
         end
@@ -54,8 +62,12 @@ module Cloudflare
         private
 
         def persist_rails_record
+          prepare_rails_record.persist
+        end
+
+        def prepare_rails_record
           require "mailbox_kit/inbound_email"
-          MailboxKit::InboundEmail.persist(source: body, message_checksum: message_checksum,
+          MailboxKit::InboundEmail.prepare(source: body, message_checksum: message_checksum,
             message_id: stable_message_id, metadata: storage_metadata)
         end
 

data/lib/cloudflare/email/version.rb

@@ -1,5 +1,5 @@
 module Cloudflare
   module Email
-    VERSION = "0.4.0"
+    VERSION = "0.4.2"
   end
 end

data/templates/deploy-to-cloudflare/README.md

@@ -3,13 +3,13 @@
 Receive email in Rails with the [cloudflare-email gem](https://github.com/cole-robertson/cloudflare-email).
 This template stores inbound mail in private R2, delivers it through a Queue,
 and retries from storage when Rails is unavailable. It includes scheduled
-recovery and Worker logs. No React app or extra server is needed.
+recovery and Worker logs.
 
 [![Deploy to Cloudflare](https://deploy.workers.cloudflare.com/button)](https://deploy.workers.cloudflare.com/?url=https://github.com/cole-robertson/cloudflare-email/tree/main/templates/deploy-to-cloudflare)
 
 ## 1. Prepare Rails
 
-Install `gem "cloudflare-email", "~> 0.3.0"` and follow the
+Install `gem "cloudflare-email", "~> 0.4.0"` and follow the
 [Rails setup guide](https://github.com/cole-robertson/cloudflare-email/blob/main/docs/getting-started.md).
 Deploy Rails with Action Mailbox storage and routing configured before connecting
 real email. Generate a secret with `openssl rand -hex 32` and configure it as
@@ -40,8 +40,8 @@ cron are present. Keep the bucket private with no lifecycle rule expiring
 
 **Choose your address shape:** `acme@in.example.com` on one receiving domain,
 or `invoices@acme.in.example.com` with dynamic organization subdomains.
-Use the [shared domain setup guide](docs/domain-setup.md) for the setup worksheet,
-Cloudflare DNS/routing steps, cited Rebulk example, and verification commands.
+Use the [shared domain setup guide](docs/domain-setup.md) for DNS,
+Worker routing, and verification commands.
 The same Worker serves both; there is no per-organization Worker allowlist.
 
 In Cloudflare, enable Email Routing for the intended receiving domain/subdomain
@@ -56,8 +56,7 @@ If using the gem's managed mailboxes, register the receiving mailbox/address in
 Rails too. Tenant support and domain catch-all receiving remain explicit choices.
 After verifying a dynamic namespace as described in the guide, register new
 organization domains in Rails without routinely adding provider rules per org.
-Wildcard DNS behavior must be verified on your Cloudflare account; the guide
-distinguishes working Rebulk evidence from Cloudflare's explicit onboarding docs.
+Verify wildcard receiving on your account before relying on unlisted subdomains.
 Outbound Email Sending credentials and delivery-event subscriptions are separate
 from this inbound Worker setup.
 
@@ -87,10 +86,9 @@ also tolerate retries. Configure operational alerts in your monitoring service.
 
 ## Updates and local development
 
-Your copied repository belongs to you; gem updates do not update its Worker
-automatically. Review upstream release notes, deploy Rails first, then port Worker
-changes into your copy while preserving your resource names, secrets, and pending
-storage. Keep production resources separate from test resources.
+Your copied repository contains your Worker deployment. Deploy changes from that
+repository; updating the gem does not deploy the Worker. Preserve resource names,
+secrets, and pending storage.
 
 Use Node 22.12+ (or a supported newer version): `npm ci`, `npm test`, and
 `npm run check`. For local development, copy `.dev.vars.example` to `.dev.vars`,