hailer 0.2.0-x86_64-linux-gnu → 0.3.0-x86_64-linux-gnu

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0a964adac0d7271e72ad3e812ddb0d4f7e8c500db607d42cd63bd2714bcb715f
-  data.tar.gz: dc46d7e0ff86ef454ff21f27f93f2149d8879ba1552d8f5b72e261f7c0963965
+  metadata.gz: 35771e1a1ff4c4269d63e539cc3e5a54729cc9236bd04343887cf2508ef1d67c
+  data.tar.gz: 2d54ed25edaa9e6d4181b30db70dae9e83534ab0eb7b175a0bf226a068740c64
 SHA512:
-  metadata.gz: 4ce86cc5d91bf6d626a9b236680131e8a6df58b7a78209ae7cfbb00acd99776a2f608a0cbaa862ff2aa349c71314f75e5df733fcf5ab7f07e9001634748145ee
-  data.tar.gz: 306ca32328b32236cbcca86358458dd2430b5676197a041cf0686cc9ed755b69c47f281ce0c6e48c948b4c7153b0fab48bbf78ff9bacfb44e90b877a6bd7fe43
+  metadata.gz: eb61ca82518a24f1d0c5e129a4a9dcf0cc8dfd8f678252f4032057e2bb912b0d5d9e8a90f716e81f11305e3845ae1ab2e3d72b0e911a0d2008035414d10cf2ce
+  data.tar.gz: 70fb56b15fd758f65d7a2d696e72d347b48153ceab571efe6ea68dde2b4e8eb5a4b8371f541494a21fba6e126f27ccee0560d52e972ab1324af62271a2268450

data/CHANGELOG.md

@@ -1,5 +1,19 @@
 # Changelog
 
+## Working tree — durable delivery (unreleased)
+
+- Opt-in `:hailer_queued` Rails adapter with stable job keys and frozen intake retries.
+- PostgreSQL ownership receipts, immutable submission replay/conflict and explicit spool budgets.
+- Supervised fenced dispatch, journaled attempts, safe retries and held unknown outcomes.
+- Tagged SES feedback correlation, recipient facts, audited cancellation and resolution.
+- Deadline/body retention, independent lifecycle feed and intake/dispatcher readiness.
+- Native Rust future traits without `async-trait`; throttle-machines 0.3.1.
+- Current-source CommonShare queued smoke with loopback SES; no production cutover.
+- New Linux binary builds target Debian 13 Trixie (glibc 2.41), with locked GNU Ruby dependency variants.
+- Opt-in pre-send address verification (Emailable first; swappable via `Verifier` enum), `[email_verification]` in `hailer.toml`, cached verdicts in `hailer.address_verifications`. Fails open by default - a verifier outage or `unknown`/`risky` result never blocks a send unless explicitly configured into `reject_states`.
+- Domain/pattern blocklist (`hailer.domain_blocks`): exact domains or `*.`-suffix globs, always active. A small hand-curated disposable-provider list ships with the repo (source `curated`, only changes by upgrading); operators add their own via `hailer ban-domain`/`unban-domain` (source `operator`). Checked before suppression and verification - the cheapest, broadest filter runs first.
+- New `quarantined` message status, queued pipeline only: a non-ASCII recipient domain, or (opt-in `[domain_intel]`, RDAP-backed) a newly registered one, holds the whole message for operator review (`hailer release-quarantine`/`cancel-message`) instead of a hard reject. Excluded from the dispatch index like `held_unknown`; expires on the normal deadline like `queued`.
+
 ## 0.2.0 (unreleased)
 
 - PostgreSQL-only live state, with atomic feedback and commit-safe cursors.

data/README.md

@@ -13,6 +13,12 @@ This repository prepares two things at version **0.2.0** (not published yet):
 ActionMailer → hailer gem → private Mothership route → hailer binary → SES
 ```
 
+The working tree also implements an **opt-in durable queued adapter**. See
+[durable delivery](docs/durable-delivery.md) for ownership, recovery, routing
+and operator controls. It is not included in the preserved 0.2 release
+candidates. Current-source macOS, ARM64 Linux and x86_64 Linux verification
+gates pass locally; this work has not been published or deployed.
+
 ## Rails
 
 Requires Ruby 4.0+ and Rails 8.1 or newer within Rails 8.x.
@@ -242,6 +248,101 @@ an interrupted send can have an unknown outcome.
 `/internal/` is a name, not an access control. Keep the bind private. The client
 does not follow redirects or use environment HTTP proxies.
 
+## Impossible addresses
+
+Always on, no configuration: `toto@example.com` is syntactically a real
+address, but RFC 2606 reserves `example.com`/`.net`/`.org` and the
+`test`/`example`/`invalid`/`localhost` TLDs for documentation - no real
+mailbox ever lives there. Rejected at the same boundary as a malformed
+address (`SendRequest::validate`), before any guard in the chain runs, for
+both the synchronous and queued adapters. Not a policy choice like the
+domain blocklist below - there is nothing to configure or an operator could
+turn off.
+
+## Domain blocklist
+
+```text
+SendRequest -> impossible-address check -> domain blocklist (free, local) -> suppression (free, local) -> verification (paid, external) -> SES
+```
+
+Always active - an empty `hailer.domain_blocks` table is a no-op, there is
+no toggle. A pattern is an exact domain (`guerrillamail.com`) or a
+`*.`-prefixed suffix glob (`*.kp` matches that domain and any subdomain).
+A small hand-curated list of known disposable-mail providers ships with the
+repo (`source = 'curated'`, migration `05_domain_blocks.sql`); it only grows
+by upgrading Hailer, never fetched from a third party at runtime. Operators
+add their own:
+
+```sh
+hailer ban-domain --pattern "*.kp" --reason "sanctioned TLD"
+hailer unban-domain --pattern "*.kp"
+```
+
+`unban-domain` only removes an `operator`-sourced pattern - a curated one
+cannot be removed this way, by design.
+
+## Address verification
+
+```text
+SendRequest -> suppression check (cheap, local) -> address verification (paid, external) -> SES
+```
+
+Opt-in pre-send deliverability check, off by default. Enable with
+`[email_verification]` in `hailer.toml`:
+
+```toml
+[email_verification]
+provider = "emailable"
+reject_states = ["undeliverable"]  # risky/unknown fail open unless added here
+cache_ttl_seconds = 604800
+timeout_seconds = 5
+```
+
+Set `EMAILABLE_API_KEY` in the process environment, never in `hailer.toml` -
+the same rule as `HAILER_DATABASE_URL` and AWS credentials. Suppression
+filters first since it is a free local lookup; verification only runs for
+recipients that survive it, so a suppressed address never costs a paid
+lookup. Verdicts are cached in `hailer.address_verifications` (a week by
+default) - a retried send against the same address reuses the cached verdict
+instead of paying for another check.
+
+A verifier outage, timeout, or an `unknown`/`risky` result all fail open: this
+middleware is never a new reason a real send stops working unless a
+deployment explicitly adds those states to `reject_states`. Adding another
+provider (ZeroBounce, NeverBounce, ...) means a new `Verifier` enum variant in
+`src/verify.rs`; nothing else in the send path changes.
+
+### Bulk pre-screening (batch)
+
+A separate concern from the real-time guard above: pre-verify a whole list
+*before* ever trying to send to it, not a per-send check. Emailable's batch
+API is async (submit, then poll) and only returns per-address results inline
+for batches up to 1,000 - larger ones get a ZIP/CSV download link this client
+deliberately does not fetch or parse, to avoid a zip/csv dependency:
+
+```sh
+hailer verify-batch --file addresses.txt   # one address per line, max 1000
+hailer batch-status --id BATCH_ID          # poll; caches each verdict once complete
+```
+
+`batch-status` writes every returned verdict into `hailer.address_verifications`
+(the same cache the real-time guard reads), so a later send to an
+already-screened address reuses the cached verdict instead of paying for
+another single-address check. A batch over 1,000 addresses is reported with
+its `download_file` URL but is not cached - split into smaller batches
+instead.
+
+`verify-batch` never submits an address that does not need it: exact
+duplicate lines (by the same normalized key the cache itself is keyed by -
+`USER+alias@x` and `user@x` are the same address), and anything already in
+`hailer.address_verifications` with an unexpired verdict, are dropped before
+the batch is ever submitted - never spending a credit on them. Every
+exclusion is counted and reported (`invalid_lines`, `duplicate_lines`,
+`already_cached`), never silently dropped. `hailer emailable-account` checks
+the Emailable account's remaining credits directly; `submit_batch` itself
+also checks this before submitting and refuses (without ever reaching the
+server) if the account has none left or fewer than the batch needs.
+
 ## Failure semantics
 
 An HTTP 200 means **SES accepted the message**, not that the recipient received
@@ -259,9 +360,9 @@ it. A successful response must include a nonblank `message_id`.
 | Connection, TLS, or timeout failure | `Hailer::TransportError` |
 | Missing/invalid endpoint or timeout settings | `Hailer::ConfigurationError` |
 
-The gem does not retry. A lost response can happen **after** SES accepted the
-message; retrying may duplicate it. There is no durable **outbound** queue or
-send-idempotency store yet; feedback deduplication does not deduplicate sends.
+The synchronous adapter does not retry. A lost response can happen **after** SES
+accepted the message; retrying may duplicate it. This route does not use the
+opt-in outbound spool or its submission-key deduplication.
 Rails jobs can also retry, so configure them deliberately.
 The SES SDK makes **one send attempt**. AWS throttling/transient failures return
 503, permanent service rejections 502, and disconnects/timeouts/missing message
@@ -305,8 +406,12 @@ test database, Mothership, the optimized binary from the macOS gem, its installe
 client, and a loopback SES fixture. Its existing AWS
 receivers remain until the real subscription and suppression cutover is verified.
 No deployed AWS routing has been changed.
-Validation/behavior plugins, a durable outbound queue, distributed account-wide
-quotas, and send idempotency remain later work. No plugin API is promised yet.
+The current-source queued adapter and dispatcher also pass a CommonShare smoke
+using an isolated Bundler Gemfile, real Mothership and loopback SES. No app
+Gemfile or production configuration was changed. Its legacy history observer
+still needs queued-ownership semantics before opt-in deployment.
+Validation/behavior plugins and distributed account-wide quotas remain later
+work. No plugin API or provider exactly-once delivery is promised.
 
 ## Development
 
@@ -354,8 +459,17 @@ in the packaging task publishes a gem.
 ## Linux releases and RubyGems
 
 The **Linux binary gems** GitHub Actions workflow builds `x86_64-linux-gnu`
-and `aarch64-linux-gnu` natively against Debian 12 (glibc 2.36). These artifacts
-require glibc 2.36 or newer; they are not Alpine/musl builds.
+and `aarch64-linux-gnu` natively against Debian 13 Trixie (glibc 2.41).
+New Trixie-built artifacts target glibc 2.41 or newer, not Bookworm or
+Alpine/musl. The preserved older 0.2 candidates retain their original baseline.
+Local OrbStack verification uses `test/linux/Dockerfile` for each architecture
+and a separate `ruby:4.0-slim-trixie` runtime.
+Use a copied checkout in separate per-architecture Docker work volumes; do not
+reuse the macOS `target/` or Ruby bundle. Run the locked Ruby tests, release Rust
+tests and `native:package` with an isolated PostgreSQL fixture, then install each
+gem in its slim runtime and exercise the launcher, migrations and import CLI.
+When transferring a macOS snapshot, use `COPYFILE_DISABLE=1 tar --no-xattrs`
+so AppleDouble `._*` metadata cannot appear as spurious SQL migrations.
 
 ```sh
 gh workflow run linux-gems.yml --repo seuros/hailer -f tag=v0.2.0

data/docs/durable-delivery.md

@@ -0,0 +1,217 @@
+# Durable delivery (in development)
+
+The queued Ruby interface, PostgreSQL spool and supervised dispatcher are
+implemented. CommonShare source-client and installed macOS binary-gem smoke
+tests pass. ARM64 and x86_64 Linux release tests, binary packaging and clean slim
+Trixie runtime checks also pass in OrbStack (x86_64 under emulation).
+Do not enable queued delivery against the existing 0.2 release
+candidates or treat this working tree as a published durable-delivery release.
+
+## Opt-in Rails configuration
+
+Keep `HAILER_URL` pointing to the existing private `/internal/mail/send` endpoint.
+The queued client uses the same validated origin with `/internal/mail/messages`.
+
+```ruby
+# config/environments/production.rb
+config.action_mailer.delivery_method = :hailer_queued
+```
+
+The initializer registers the adapter and selects `Hailer::MailDeliveryJob` for
+this opt-in configuration. The default remains the synchronous `:hailer`
+adapter, whose HTTP requests never retry.
+
+For a per-mailer opt-in, select both:
+
+```ruby
+self.delivery_method = :hailer_queued
+self.delivery_job = Hailer::MailDeliveryJob
+```
+
+An explicitly configured application delivery job is not replaced. It must
+inherit `Hailer::MailDeliveryJob` or include `Hailer::DeliveryJobContext` to
+obtain stable submission keys across job retries. This uses public ActiveJob
+callbacks, not Rails monkey patches.
+
+## Identities and retries
+
+- Standard queued jobs use job ID plus delivery ordinal. Different jobs are
+  different intents unless the application supplies a business key.
+- Direct `deliver_now` calls generate a fresh key per invocation.
+- Supply a business key through `delivery_method_options` when separate calls
+  represent the same intended message:
+
+  ```ruby
+  mail(
+    to: recipient,
+    subject: 'Invoice',
+    delivery_method_options: { submission_key: "invoice:#{invoice.id}:issued" }
+  )
+  ```
+
+- Supply an optional ISO 8601 `expires_at` through those same options. The
+  service contract defaults to 24 hours and allows shorter deadlines.
+- A queued intake call serializes its command once. Up to three HTTP attempts
+  reuse those exact bytes, key and deadline after transport loss or retryable
+  admission errors (429/503). Set `intake_attempts: 1` to disable these retries.
+- Validation, submission conflicts, malformed responses and provider-unknown
+  errors do not retry.
+- Rails re-renders mail when a job is performed again. If the bytes or envelope
+  change under an accepted key, the service must return a conflict; the adapter
+  raises `Hailer::SubmissionConflictError`. Job keys alone do not provide
+  transparent exactly-once business delivery.
+
+Submission keys are command metadata, not MIME headers.
+
+## Private status and lifecycle feed
+
+```ruby
+client = Hailer::LifecycleClient.new # same HAILER_URL origin
+status = client.status(hailer_message_id)
+page = client.poll(after: saved_cursor, limit: 100)
+```
+
+Status distinguishes the original RFC Message-ID, Hailer ID and provider ID,
+and never returns MIME. `status.attempts` and `status.recipient_outcomes` contain
+the latest 100 records each, in chronological order. Their corresponding
+`*_truncated` flags report older records; use the lifecycle feed for complete
+observations. An accepted message can still have an uncertain historical
+attempt after an operator-authorized retry. Lifecycle events have their own cursor space; do not
+reuse an existing `FeedbackClient` cursor. Persist `page.next_cursor` only after
+processing the page successfully. Re-reading a cursor does not delete events.
+
+The service's `[queued]` configuration requires explicit `max_bytes` and
+`max_messages`. Bytes measure canonical logical command payload, not PostgreSQL
+table/index/WAL/backups or a physical disk quota. All replicas of a deployment
+must use the same budgets. Admission is transactional; a replay does not consume
+another slot, and accepted messages are never evicted to admit new ones.
+
+## Acceptance is not delivery
+
+HTTP 202 means the service durably owns the message. The adapter exposes its ID
+through `message.header[:hailer_message_id]` before Rails observers run. It does
+not fabricate `ses_message_id`; provider acceptance and recipient delivery
+are later observations.
+
+Application observers must distinguish durable ownership from provider
+acceptance: persist the Hailer ID and an owned/queued state, then consume
+lifecycle facts separately. CommonShare's existing observer still writes
+`status: "sent"` and only the SES ID for every delivered callback. The isolated
+queued smoke verifies transport compatibility, not that observer's readiness
+for production opt-in. Keep its synchronous configuration until this application
+cutover work is done.
+
+Unknown provider outcomes are held without an automatic resend. Seven-day
+unknown-body retention (measured from durable intake) does not extend the
+24-hour send-start deadline. Body purge retains receipts and uncertainty.
+Only positive evidence of no acceptance permits automatic retry, with jittered
+backoff capped at 600 seconds and at most 255 automatic attempts per intent.
+Expiry stops new attempts; it never proves a previous attempt was unsent.
+
+Live bodies are purged after confirmed acceptance, terminal rejection or
+suppression, cancellation, or safe queued expiry. Purge releases logical
+capacity, not PostgreSQL allocated disk, backups or copies already in-flight.
+
+## Feedback and private routing
+
+Set root `ses_configuration_set = "hailer-feedback"` in Hailer's TOML, or use
+an equivalent default configuration set on the SES sending identity. Configure
+SES event publishing through the dedicated SNS/SQS feedback route, preserving
+`mail.tags.hailer_message_id` and `mail.tags.hailer_attempt_id`. These unique tags
+allow feedback to resolve an unknown SDK response. A provider ID already
+captured from an accepted response also correlates later untagged feedback.
+Without tags and without a captured provider ID, feedback cannot resolve
+uncertainty. Hailer never guesses from recipient, timestamps or RFC Message-ID.
+Feedback received before an untagged provider ID is captured is not retroactively
+correlated; retain the tags for reliable recovery.
+
+Mothership must route the new private paths as well as the old synchronous
+ones. A suitable bind pattern is:
+
+```toml
+routes = [{ bind = "mail", pattern = "^/internal/mail/(send|events|health|ready|intake-ready|dispatch-ready|messages|messages/events|messages/[0-9a-f-]{36})$" }]
+```
+
+Keep this bind private. Intake readiness depends on durable writable capacity,
+not SES health. `/internal/mail/dispatch-ready` separately reports worker,
+bootstrap and control eligibility. Neither readiness response guarantees that
+a later send will succeed.
+
+## Operator control
+
+Run against the intended database using the same `HAILER_DATABASE_URL`:
+
+```sh
+hailer message-status --id MESSAGE_UUID
+hailer cancel-message --id MESSAGE_UUID --reason "intent withdrawn"
+hailer resolve-message --id MESSAGE_UUID --outcome accepted \
+  --provider-message-id VERIFIED_SES_ID --reason "verified provider evidence"
+hailer resolve-message --id MESSAGE_UUID --outcome not-accepted \
+  --reason "verified no provider acceptance"
+hailer resolve-message --id MESSAGE_UUID --outcome retry \
+  --acknowledge-duplicate-risk --reason "investigated; accept duplicate risk"
+```
+
+Cancellation is only possible before dispatch and cannot unsend mail.
+Resolution defaults to the current held-unknown attempt. Use `--attempt-id
+ATTEMPT_UUID` to resolve an older unknown journal entry without overwriting a
+later attempt. Retry is allowed only for the current held intent, with an
+available body, an unexpired deadline, no acceptance latch and explicit
+duplicate-risk acknowledgement. A new attempt never erases the old uncertainty.
+All decisions require an audit reason. After expiry or body purge, sending
+requires a new intent/key.
+
+## Quarantine
+
+Opt-in, queued pipeline only - a synchronous `:hailer` send has no way to
+hold a message for later review, so a non-ASCII domain or a newly
+registered one makes the synchronous adapter reject outright. The queued
+adapter holds instead:
+
+```toml
+[domain_intel]
+provider = "rdap"
+new_domain_threshold_days = 30
+```
+
+A recipient domain that is not plain ASCII (a malformed/unencoded IDN, or a
+homograph-spoofing attempt - legitimate IDNs are transmitted as ASCII
+punycode) is always checked, regardless of this block. `[domain_intel]`
+additionally checks the domain's RDAP registration date; newer than the
+threshold holds the message as `quarantined` instead of admitting it as
+`queued`. The whole message is quarantined, not just the flagged recipient -
+this is an operator review gate on intent, not a per-recipient filter like
+suppression or the domain blocklist (`hailer ban-domain`). RDAP's bootstrap
+registry is public; no credential is needed.
+
+```sh
+hailer release-quarantine --id MESSAGE_UUID --reason "reviewed, legitimate"
+hailer cancel-message --id MESSAGE_UUID --reason "rejected on review"
+```
+
+Releasing resets `next_attempt_at` to now, so the message re-enters the
+normal dispatch queue immediately. `cancel-message` also accepts a
+quarantined message. An unactioned quarantine still expires on the same
+deadline as a normal `queued` message.
+
+## Local application verification
+
+`test/commonshare_smoke.rb` runs inside CommonShare's test environment with a
+real Mothership bay and loopback SES, never real AWS credentials. Use a fresh
+loopback PostgreSQL database named `hailer_smoke_*` for each queued run.
+The script intentionally reuses one business key to check replay after purge.
+
+```sh
+RAILS_ENV=test DISABLE_BOOTSNAP=1 \
+HAILER_SMOKE_QUEUED=1 \
+HAILER_SMOKE_BINARY=/absolute/path/to/current/hailer \
+HAILER_DATABASE_URL=postgres://...@127.0.0.1:5432/hailer_smoke_queued \
+bundle exec rails runner /absolute/path/to/hailer/test/commonshare_smoke.rb
+```
+
+An isolated temporary `BUNDLE_GEMFILE` can select the working-tree gem or locally
+installed verification gem without editing CommonShare's Gemfile or lock.
+Set `HAILER_SMOKE_EXPECTED_PLATFORM=arm64-darwin` to assert native-gem selection.
+Omit `HAILER_SMOKE_QUEUED` for the unchanged synchronous contract. The test holds
+the provider response until Rails has its receipt, then checks the journal,
+correlation tags, envelope/attachment, body purge, replay, conflict and feeds.

data/hailer.example.toml

@@ -3,6 +3,11 @@ threads = 2
 database_pool_size = 5
 shutdown_timeout_seconds = 30
 
+# Optional SES event-publishing configuration set. Its events must include mail
+# tags and reach the dedicated SNS/SQS feedback route to resolve unknown sends.
+# Otherwise configure an appropriate default set on the sending identity.
+# ses_configuration_set = "hailer-feedback"
+
 # Opt in: initial AWS account/region sync gates send readiness.
 [suppression_sync]
 interval_seconds = 3600
@@ -12,7 +17,33 @@ interval_seconds = 3600
 # recipients_per_second = 10.0
 # burst = 50
 
+# In development: durable intake requires explicit logical payload budgets.
+# Requires the durable-dispatch build, not the existing 0.2 release candidates.
+# [queued]
+# max_bytes = 268435456
+# max_messages = 10000
+# unknown_retention_seconds = 604800
+
 [feedback]
 # Dedicated queue; subscribe the SNS topic with RawMessageDelivery=false.
 queue_url = "https://sqs.us-east-1.amazonaws.com/123456789012/hailer-feedback"
 topic_arn = "arn:aws:sns:us-east-1:123456789012:mail-feedback"
+
+# Opt in: pre-send deliverability check (Emailable today; the provider is a
+# swappable enum, not hardcoded). Set EMAILABLE_API_KEY separately - never
+# here. Verdicts are cached (hailer.address_verifications) so a retry of the
+# same address does not pay for a second lookup.
+# [email_verification]
+# provider = "emailable"
+# reject_states = ["undeliverable"]  # risky/unknown fail open by default
+# cache_ttl_seconds = 604800
+# timeout_seconds = 5
+
+# Opt in, queued pipeline only: hold a message for operator review
+# (hailer release-quarantine / cancel-message) instead of a hard reject.
+# A non-ASCII recipient domain is always checked regardless of this block;
+# this only adds the RDAP new-domain check. No credential needed - RDAP's
+# bootstrap registry is public.
+# [domain_intel]
+# provider = "rdap"
+# new_domain_threshold_days = 30

data/lib/hailer/delivery_context.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+require 'active_support'
+require 'active_support/current_attributes'
+
+module Hailer
+  # Scoped by the public ActiveJob callback, restored even on exceptions and
+  # nested jobs. No submission key is injected into the rendered MIME.
+  class DeliveryContext < ActiveSupport::CurrentAttributes
+    attribute :job_id, :ordinal
+
+    def next_submission_key
+      return unless job_id
+
+      self.ordinal = ordinal.to_i + 1
+      "hailer:job:#{job_id}:#{ordinal}"
+    end
+  end
+end

data/lib/hailer/delivery_method.rb

@@ -11,6 +11,14 @@ module Hailer
     end
 
     def deliver!(message)
+      Client.new(**settings).deliver(payload_for(message)).tap do |response|
+        message.header[:ses_message_id] = response.message_id
+      end
+    end
+
+    private
+
+    def payload_for(message)
       payload = {
         raw_mime: message.to_s,
         to_addresses: Array(explicit_envelope?(message, :to) ? message.smtp_envelope_to : message.to),
@@ -23,13 +31,9 @@ module Hailer
         payload[:from_email_address] = message.smtp_envelope_from
       end
 
-      Client.new(**settings).deliver(payload).tap do |response|
-        message.header[:ses_message_id] = response.message_id
-      end
+      payload
     end
 
-    private
-
     # Match the existing SESv2 adapter's distinction between Mail's computed
     # defaults and explicitly assigned SMTP envelope addresses.
     def explicit_envelope?(message, field)

data/lib/hailer/errors.rb

@@ -19,6 +19,7 @@ module Hailer
   end
 
   class ValidationError < DeliveryError; end
+  class SubmissionConflictError < DeliveryError; end
   class RejectedError < DeliveryError; end
   class UnavailableError < DeliveryError; end
   class RateLimitedError < DeliveryError; end

data/lib/hailer/lifecycle_client.rb

@@ -0,0 +1,124 @@
+# frozen_string_literal: true
+
+require 'time'
+
+module Hailer
+  MessageStatus = Data.define(
+    :hailer_message_id, :state, :version, :body_present, :provider_message_id,
+    :rfc_message_id, :accepted_at, :expires_at, :attempts, :recipient_outcomes,
+    :attempts_truncated, :recipient_outcomes_truncated
+  )
+  MessageEvent = Data.define(:sequence, :hailer_message_id, :kind, :occurred_at, :details)
+  MessageEventPage = Data.define(:events, :next_cursor)
+
+  # Deliberately separate from the unchanged provider-feedback client/feed.
+  class LifecycleClient
+    ID = /\A[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\z/
+    STATES = %w[queued claimed dispatching retry_waiting held_unknown accepted rejected suppressed cancelled expired].freeze
+    MAX_RESPONSE_BYTES = 2 * 1024 * 1024
+
+    def initialize(**settings)
+      @transport = Transport.new(**settings)
+    end
+
+    def status(hailer_message_id)
+      unless hailer_message_id.is_a?(String) && ID.match?(hailer_message_id)
+        raise ConfigurationError, 'expected a Hailer message UUID'
+      end
+      document = @transport.get("/internal/mail/messages/#{hailer_message_id}",
+                                max_response_bytes: MAX_RESPONSE_BYTES)
+      receipt = document['receipt']
+      unless receipt.is_a?(Hash) && receipt['hailer_message_id'] == hailer_message_id &&
+             STATES.include?(document['status']) &&
+             document['version'].is_a?(Integer) && document['version'] >= 0 &&
+             [true, false].include?(document['body_present']) &&
+             optional_text?(document['provider_message_id']) && optional_text?(document['rfc_message_id'])
+        raise ProtocolError, 'invalid Hailer message status'
+      end
+      accepted_at = timestamp(receipt['accepted_at'])
+      expires_at = timestamp(receipt['expires_at'])
+      raise ProtocolError, 'invalid Hailer message deadline' unless expires_at > accepted_at
+
+      MessageStatus.new(
+        hailer_message_id: hailer_message_id.dup.freeze, state: document['status'].freeze,
+        version: document['version'], body_present: document['body_present'],
+        provider_message_id: document['provider_message_id']&.freeze,
+        rfc_message_id: document['rfc_message_id']&.freeze,
+        accepted_at: accepted_at, expires_at: expires_at,
+        attempts: status_records(document, 'attempts'),
+        recipient_outcomes: status_records(document, 'recipient_outcomes'),
+        attempts_truncated: truncation_flag(document, 'attempts_truncated'),
+        recipient_outcomes_truncated: truncation_flag(document, 'recipient_outcomes_truncated')
+      )
+    end
+
+    def poll(after:, limit: 100)
+      unless after.is_a?(Integer) && (0..9_223_372_036_854_775_807).cover?(after) &&
+             limit.is_a?(Integer) && (1..100).cover?(limit)
+        raise ConfigurationError, 'lifecycle requires a supported nonnegative cursor and limit in 1..100'
+      end
+      document = @transport.post('/internal/mail/messages/events', { after: after, limit: limit },
+                                 max_response_bytes: MAX_RESPONSE_BYTES)
+      entries = document['events']
+      unless entries.is_a?(Array) && entries.size <= limit && document['next_cursor'].is_a?(Integer)
+        raise ProtocolError, 'invalid Hailer lifecycle page'
+      end
+      previous = after
+      events = entries.map do |entry|
+        unless entry.is_a?(Hash) && entry['sequence'].is_a?(Integer) && entry['sequence'] > previous
+          raise ProtocolError, 'lifecycle sequences must increase past the requested cursor'
+        end
+        event = entry['event']
+        unless event.is_a?(Hash) && event['hailer_message_id'].is_a?(String) &&
+               ID.match?(event['hailer_message_id']) && text?(event['kind']) && event['details'].is_a?(Hash)
+          raise ProtocolError, 'invalid Hailer lifecycle event'
+        end
+        previous = entry['sequence']
+        MessageEvent.new(
+          sequence: previous, hailer_message_id: event['hailer_message_id'].freeze,
+          kind: event['kind'].freeze, occurred_at: timestamp(event['occurred_at']),
+          details: freeze_json(event['details'])
+        )
+      end
+      raise ProtocolError, 'lifecycle cursor does not match its page' unless document['next_cursor'] == previous
+
+      MessageEventPage.new(events: events.freeze, next_cursor: previous)
+    end
+
+    private
+
+    def text?(value) = value.is_a?(String) && !value.strip.empty?
+    def optional_text?(value) = value.nil? || text?(value)
+
+    def status_records(document, name)
+      records = document.fetch(name, [])
+      unless records.is_a?(Array) && records.size <= 100 && records.all? { |record| record.is_a?(Hash) }
+        raise ProtocolError, 'invalid Hailer status observations'
+      end
+      freeze_json(records)
+    end
+
+    def truncation_flag(document, name)
+      flag = document.fetch(name, false)
+      raise ProtocolError, 'invalid Hailer status truncation flag' unless [true, false].include?(flag)
+
+      flag
+    end
+
+    def timestamp(value)
+      raise ProtocolError, 'invalid Hailer lifecycle timestamp' unless text?(value)
+
+      Time.iso8601(value).freeze
+    rescue ArgumentError
+      raise ProtocolError, 'invalid Hailer lifecycle timestamp'
+    end
+
+    def freeze_json(value)
+      case value
+      when Hash then value.each_value { |child| freeze_json(child) }
+      when Array then value.each { |child| freeze_json(child) }
+      end
+      value.freeze
+    end
+  end
+end

data/lib/hailer/mail_delivery_job.rb

@@ -0,0 +1,18 @@
+# frozen_string_literal: true
+
+module Hailer
+  # Custom application delivery jobs can include this public callback module.
+  module DeliveryJobContext
+    extend ActiveSupport::Concern
+
+    included do
+      around_perform do |job, perform|
+        DeliveryContext.set(job_id: job.job_id, ordinal: 0) { perform.call }
+      end
+    end
+  end
+
+  class MailDeliveryJob < ActionMailer::MailDeliveryJob
+    include DeliveryJobContext
+  end
+end

data/lib/hailer/queued_client.rb

@@ -0,0 +1,53 @@
+# frozen_string_literal: true
+
+module Hailer
+  Acceptance = Data.define(:hailer_message_id)
+
+  # Acceptance means PostgreSQL owns the message, not that SES accepted it.
+  class QueuedClient
+    PATH = '/internal/mail/messages'
+
+    def initialize(intake_attempts: 3, **settings)
+      unless intake_attempts.is_a?(Integer) && (1..3).cover?(intake_attempts)
+        raise ConfigurationError, 'intake_attempts must be in 1..3'
+      end
+      @intake_attempts = intake_attempts
+      @transport = Transport.new(**settings)
+    end
+
+    def submit(payload, submission_key:, expires_at: nil)
+      unless submission_key.is_a?(String) && !submission_key.strip.empty? && submission_key.bytesize <= 256
+        raise ConfigurationError, 'submission_key must be nonempty text of at most 256 bytes'
+      end
+      unless payload.is_a?(Hash) && payload.keys.none? { |key| %w[submission_key expires_at].include?(key.to_s) }
+        raise ConfigurationError, 'pass submission_key and expires_at separately from the mail payload'
+      end
+      command = payload.merge(submission_key: submission_key)
+      command[:expires_at] = expires_at unless expires_at.nil?
+      # Freeze serialization before the first request: retries must not re-render
+      # MIME, regenerate keys, extend deadlines or observe caller mutations.
+      body = JSON.generate(command).freeze
+      document = submit_frozen(body)
+      id = document['hailer_message_id']
+      unless id.is_a?(String) && !id.strip.empty?
+        raise ProtocolError, 'Hailer accepted the request without a hailer_message_id'
+      end
+      Acceptance.new(hailer_message_id: id.freeze)
+    end
+
+    private
+
+    def submit_frozen(body)
+      attempts = 0
+      begin
+        attempts += 1
+        @transport.post_serialized(PATH, body, expected_status: 202)
+      rescue TransportError, UnavailableError, RateLimitedError
+        raise if attempts >= @intake_attempts
+
+        sleep(0.05 * (2**(attempts - 1)))
+        retry
+      end
+    end
+  end
+end

data/lib/hailer/queued_delivery_method.rb

@@ -0,0 +1,18 @@
+# frozen_string_literal: true
+
+require 'securerandom'
+
+module Hailer
+  class QueuedDeliveryMethod < DeliveryMethod
+    def deliver!(message)
+      job_key = DeliveryContext.next_submission_key
+      key = settings.fetch(:submission_key) { job_key || SecureRandom.uuid }
+      options = settings.except(:submission_key, :expires_at)
+      QueuedClient.new(**options).submit(
+        payload_for(message), submission_key: key, expires_at: settings[:expires_at]
+      ).tap do |receipt|
+        message.header[:hailer_message_id] = receipt.hailer_message_id
+      end
+    end
+  end
+end

data/lib/hailer/railtie.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require 'action_mailer/railtie'
+require_relative 'mail_delivery_job'
 
 module Hailer
   class Railtie < ::Rails::Railtie
@@ -10,9 +11,13 @@ module Hailer
     config.hailer = ActiveSupport::OrderedOptions.new
     config.hailer.feedback_handler = nil
 
-    initializer 'hailer.action_mailer', before: 'action_mailer.set_configs' do
+    initializer 'hailer.action_mailer', before: 'action_mailer.set_configs' do |app|
+      if app.config.action_mailer.delivery_method == :hailer_queued
+        app.config.action_mailer.delivery_job ||= 'Hailer::MailDeliveryJob'
+      end
       ActiveSupport.on_load(:action_mailer) do
         add_delivery_method :hailer, Hailer::DeliveryMethod
+        add_delivery_method :hailer_queued, Hailer::QueuedDeliveryMethod
       end
     end
   end

data/lib/hailer/transport.rb

@@ -10,7 +10,7 @@ module Hailer
     DEFAULTS = { open_timeout: 5, read_timeout: 30, write_timeout: 30 }.freeze
     MAX_RESPONSE_BYTES = 65_536
     HTTP_ERRORS = {
-      400 => ValidationError, 422 => ValidationError, 429 => RateLimitedError,
+      400 => ValidationError, 409 => SubmissionConflictError, 422 => ValidationError, 429 => RateLimitedError,
       502 => RejectedError, 503 => UnavailableError
     }.freeze
 
@@ -32,23 +32,37 @@ module Hailer
     end
 
     def post(path, payload, max_response_bytes: MAX_RESPONSE_BYTES)
+      post_serialized(path, JSON.generate(payload), max_response_bytes: max_response_bytes)
+    end
+
+    # Queued intake owns retries and passes the same serialized bytes each time.
+    # Synchronous sends continue to require HTTP 200 and never retry.
+    def post_serialized(path, body, expected_status: 200, max_response_bytes: MAX_RESPONSE_BYTES)
       request = Net::HTTP::Post.new(path)
       request.content_type = 'application/json'
+      request.body = body
+      exchange_document(request, expected_status, max_response_bytes)
+    end
+
+    def get(path, max_response_bytes: MAX_RESPONSE_BYTES)
+      exchange_document(Net::HTTP::Get.new(path), 200, max_response_bytes)
+    end
+
+    private
+
+    def exchange_document(request, expected_status, max_response_bytes)
       request['Accept'] = 'application/json'
-      request.body = JSON.generate(payload)
       http = Net::HTTP.new(@uri.hostname, @uri.port, nil)
       http.use_ssl = @uri.scheme == 'https'
       http.max_retries = 0
       @timeouts.each { |name, value| http.public_send(:"#{name}=", value) }
       status, body = exchange(http, request, max_response_bytes)
-      decode_response(status, body)
+      decode_response(status, body, expected_status)
     rescue Timeout::Error, IOError, EOFError, SystemCallError, SocketError,
            OpenSSL::SSL::SSLError, Net::HTTPBadResponse, Net::HTTPHeaderSyntaxError => error
       raise TransportError, "Hailer transport failed (#{error.class}); delivery outcome may be unknown"
     end
 
-    private
-
     def exchange(http, request, limit)
       status = nil
       body = +''
@@ -65,11 +79,11 @@ module Hailer
       [status, body]
     end
 
-    def decode_response(status, body)
+    def decode_response(status, body, expected_status)
       document = JSON.parse(body)
       raise ProtocolError, 'Hailer response must be a JSON object' unless document.is_a?(Hash)
 
-      unless status == 200
+      unless status == expected_status
         message = document['error']
         message = "Hailer returned HTTP #{status}" unless message.is_a?(String) && !message.empty?
         if status == 503 && document['code'] == 'outcome_unknown'

data/lib/hailer/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Hailer
-  VERSION = '0.2.0'
+  VERSION = '0.3.0'
 end

data/lib/hailer.rb

@@ -5,6 +5,10 @@ require_relative 'hailer/errors'
 require_relative 'hailer/transport'
 require_relative 'hailer/client'
 require_relative 'hailer/delivery_method'
+require_relative 'hailer/queued_client'
+require_relative 'hailer/delivery_context'
+require_relative 'hailer/queued_delivery_method'
+require_relative 'hailer/lifecycle_client'
 require_relative 'hailer/feedback_client'
 require_relative 'hailer/feedback_consumer'
 require_relative 'hailer/railtie'

data/vendor/x86_64-linux-gnu/manifest.json

@@ -1,6 +1,6 @@
 {
-  "version": "0.2.0",
+  "version": "0.3.0",
   "platform": "x86_64-linux-gnu",
   "target": "x86_64-unknown-linux-gnu",
-  "sha256": "582548c40fdf336e1fac3db860be8e15d81ae9727b1ef8dc11be278fbae3db95"
+  "sha256": "7d4cf65da875bbcfd390069d0485798f9ebb6d55aa34b7ba2dd22ec70a0b80cb"
 }

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: hailer
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.3.0
 platform: x86_64-linux-gnu
 authors:
 - Abdelkader Boudih
@@ -149,15 +149,21 @@ files:
 - CHANGELOG.md
 - LICENSE.txt
 - README.md
+- docs/durable-delivery.md
 - exe/hailer
 - hailer.example.toml
 - lib/hailer.rb
 - lib/hailer/binary.rb
 - lib/hailer/client.rb
+- lib/hailer/delivery_context.rb
 - lib/hailer/delivery_method.rb
 - lib/hailer/errors.rb
 - lib/hailer/feedback_client.rb
 - lib/hailer/feedback_consumer.rb
+- lib/hailer/lifecycle_client.rb
+- lib/hailer/mail_delivery_job.rb
+- lib/hailer/queued_client.rb
+- lib/hailer/queued_delivery_method.rb
 - lib/hailer/railtie.rb
 - lib/hailer/transport.rb
 - lib/hailer/version.rb