hailer 0.1.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +24 -1
- data/README.md +299 -32
- data/docs/durable-delivery.md +217 -0
- data/hailer.example.toml +43 -2
- data/lib/hailer/delivery_context.rb +19 -0
- data/lib/hailer/delivery_method.rb +9 -5
- data/lib/hailer/errors.rb +4 -0
- data/lib/hailer/lifecycle_client.rb +124 -0
- data/lib/hailer/mail_delivery_job.rb +18 -0
- data/lib/hailer/queued_client.rb +53 -0
- data/lib/hailer/queued_delivery_method.rb +18 -0
- data/lib/hailer/railtie.rb +6 -1
- data/lib/hailer/transport.rb +24 -7
- data/lib/hailer/version.rb +1 -1
- data/lib/hailer.rb +4 -0
- metadata +7 -3
- data/docs/delivery-contract.md +0 -59
- data/docs/feedback.md +0 -160
|
@@ -0,0 +1,217 @@
|
|
|
1
|
+
# Durable delivery (in development)
|
|
2
|
+
|
|
3
|
+
The queued Ruby interface, PostgreSQL spool and supervised dispatcher are
|
|
4
|
+
implemented. CommonShare source-client and installed macOS binary-gem smoke
|
|
5
|
+
tests pass. ARM64 and x86_64 Linux release tests, binary packaging and clean slim
|
|
6
|
+
Trixie runtime checks also pass in OrbStack (x86_64 under emulation).
|
|
7
|
+
Do not enable queued delivery against the existing 0.2 release
|
|
8
|
+
candidates or treat this working tree as a published durable-delivery release.
|
|
9
|
+
|
|
10
|
+
## Opt-in Rails configuration
|
|
11
|
+
|
|
12
|
+
Keep `HAILER_URL` pointing to the existing private `/internal/mail/send` endpoint.
|
|
13
|
+
The queued client uses the same validated origin with `/internal/mail/messages`.
|
|
14
|
+
|
|
15
|
+
```ruby
|
|
16
|
+
# config/environments/production.rb
|
|
17
|
+
config.action_mailer.delivery_method = :hailer_queued
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
The initializer registers the adapter and selects `Hailer::MailDeliveryJob` for
|
|
21
|
+
this opt-in configuration. The default remains the synchronous `:hailer`
|
|
22
|
+
adapter, whose HTTP requests never retry.
|
|
23
|
+
|
|
24
|
+
For a per-mailer opt-in, select both:
|
|
25
|
+
|
|
26
|
+
```ruby
|
|
27
|
+
self.delivery_method = :hailer_queued
|
|
28
|
+
self.delivery_job = Hailer::MailDeliveryJob
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
An explicitly configured application delivery job is not replaced. It must
|
|
32
|
+
inherit `Hailer::MailDeliveryJob` or include `Hailer::DeliveryJobContext` to
|
|
33
|
+
obtain stable submission keys across job retries. This uses public ActiveJob
|
|
34
|
+
callbacks, not Rails monkey patches.
|
|
35
|
+
|
|
36
|
+
## Identities and retries
|
|
37
|
+
|
|
38
|
+
- Standard queued jobs use job ID plus delivery ordinal. Different jobs are
|
|
39
|
+
different intents unless the application supplies a business key.
|
|
40
|
+
- Direct `deliver_now` calls generate a fresh key per invocation.
|
|
41
|
+
- Supply a business key through `delivery_method_options` when separate calls
|
|
42
|
+
represent the same intended message:
|
|
43
|
+
|
|
44
|
+
```ruby
|
|
45
|
+
mail(
|
|
46
|
+
to: recipient,
|
|
47
|
+
subject: 'Invoice',
|
|
48
|
+
delivery_method_options: { submission_key: "invoice:#{invoice.id}:issued" }
|
|
49
|
+
)
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
- Supply an optional ISO 8601 `expires_at` through those same options. The
|
|
53
|
+
service contract defaults to 24 hours and allows shorter deadlines.
|
|
54
|
+
- A queued intake call serializes its command once. Up to three HTTP attempts
|
|
55
|
+
reuse those exact bytes, key and deadline after transport loss or retryable
|
|
56
|
+
admission errors (429/503). Set `intake_attempts: 1` to disable these retries.
|
|
57
|
+
- Validation, submission conflicts, malformed responses and provider-unknown
|
|
58
|
+
errors do not retry.
|
|
59
|
+
- Rails re-renders mail when a job is performed again. If the bytes or envelope
|
|
60
|
+
change under an accepted key, the service must return a conflict; the adapter
|
|
61
|
+
raises `Hailer::SubmissionConflictError`. Job keys alone do not provide
|
|
62
|
+
transparent exactly-once business delivery.
|
|
63
|
+
|
|
64
|
+
Submission keys are command metadata, not MIME headers.
|
|
65
|
+
|
|
66
|
+
## Private status and lifecycle feed
|
|
67
|
+
|
|
68
|
+
```ruby
|
|
69
|
+
client = Hailer::LifecycleClient.new # same HAILER_URL origin
|
|
70
|
+
status = client.status(hailer_message_id)
|
|
71
|
+
page = client.poll(after: saved_cursor, limit: 100)
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
Status distinguishes the original RFC Message-ID, Hailer ID and provider ID,
|
|
75
|
+
and never returns MIME. `status.attempts` and `status.recipient_outcomes` contain
|
|
76
|
+
the latest 100 records each, in chronological order. Their corresponding
|
|
77
|
+
`*_truncated` flags report older records; use the lifecycle feed for complete
|
|
78
|
+
observations. An accepted message can still have an uncertain historical
|
|
79
|
+
attempt after an operator-authorized retry. Lifecycle events have their own cursor space; do not
|
|
80
|
+
reuse an existing `FeedbackClient` cursor. Persist `page.next_cursor` only after
|
|
81
|
+
processing the page successfully. Re-reading a cursor does not delete events.
|
|
82
|
+
|
|
83
|
+
The service's `[queued]` configuration requires explicit `max_bytes` and
|
|
84
|
+
`max_messages`. Bytes measure canonical logical command payload, not PostgreSQL
|
|
85
|
+
table/index/WAL/backups or a physical disk quota. All replicas of a deployment
|
|
86
|
+
must use the same budgets. Admission is transactional; a replay does not consume
|
|
87
|
+
another slot, and accepted messages are never evicted to admit new ones.
|
|
88
|
+
|
|
89
|
+
## Acceptance is not delivery
|
|
90
|
+
|
|
91
|
+
HTTP 202 means the service durably owns the message. The adapter exposes its ID
|
|
92
|
+
through `message.header[:hailer_message_id]` before Rails observers run. It does
|
|
93
|
+
not fabricate `ses_message_id`; provider acceptance and recipient delivery
|
|
94
|
+
are later observations.
|
|
95
|
+
|
|
96
|
+
Application observers must distinguish durable ownership from provider
|
|
97
|
+
acceptance: persist the Hailer ID and an owned/queued state, then consume
|
|
98
|
+
lifecycle facts separately. CommonShare's existing observer still writes
|
|
99
|
+
`status: "sent"` and only the SES ID for every delivered callback. The isolated
|
|
100
|
+
queued smoke verifies transport compatibility, not that observer's readiness
|
|
101
|
+
for production opt-in. Keep its synchronous configuration until this application
|
|
102
|
+
cutover work is done.
|
|
103
|
+
|
|
104
|
+
Unknown provider outcomes are held without an automatic resend. Seven-day
|
|
105
|
+
unknown-body retention (measured from durable intake) does not extend the
|
|
106
|
+
24-hour send-start deadline. Body purge retains receipts and uncertainty.
|
|
107
|
+
Only positive evidence of no acceptance permits automatic retry, with jittered
|
|
108
|
+
backoff capped at 600 seconds and at most 255 automatic attempts per intent.
|
|
109
|
+
Expiry stops new attempts; it never proves a previous attempt was unsent.
|
|
110
|
+
|
|
111
|
+
Live bodies are purged after confirmed acceptance, terminal rejection or
|
|
112
|
+
suppression, cancellation, or safe queued expiry. Purge releases logical
|
|
113
|
+
capacity, not PostgreSQL allocated disk, backups or copies already in-flight.
|
|
114
|
+
|
|
115
|
+
## Feedback and private routing
|
|
116
|
+
|
|
117
|
+
Set root `ses_configuration_set = "hailer-feedback"` in Hailer's TOML, or use
|
|
118
|
+
an equivalent default configuration set on the SES sending identity. Configure
|
|
119
|
+
SES event publishing through the dedicated SNS/SQS feedback route, preserving
|
|
120
|
+
`mail.tags.hailer_message_id` and `mail.tags.hailer_attempt_id`. These unique tags
|
|
121
|
+
allow feedback to resolve an unknown SDK response. A provider ID already
|
|
122
|
+
captured from an accepted response also correlates later untagged feedback.
|
|
123
|
+
Without tags and without a captured provider ID, feedback cannot resolve
|
|
124
|
+
uncertainty. Hailer never guesses from recipient, timestamps or RFC Message-ID.
|
|
125
|
+
Feedback received before an untagged provider ID is captured is not retroactively
|
|
126
|
+
correlated; retain the tags for reliable recovery.
|
|
127
|
+
|
|
128
|
+
Mothership must route the new private paths as well as the old synchronous
|
|
129
|
+
ones. A suitable bind pattern is:
|
|
130
|
+
|
|
131
|
+
```toml
|
|
132
|
+
routes = [{ bind = "mail", pattern = "^/internal/mail/(send|events|health|ready|intake-ready|dispatch-ready|messages|messages/events|messages/[0-9a-f-]{36})$" }]
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
Keep this bind private. Intake readiness depends on durable writable capacity,
|
|
136
|
+
not SES health. `/internal/mail/dispatch-ready` separately reports worker,
|
|
137
|
+
bootstrap and control eligibility. Neither readiness response guarantees that
|
|
138
|
+
a later send will succeed.
|
|
139
|
+
|
|
140
|
+
## Operator control
|
|
141
|
+
|
|
142
|
+
Run against the intended database using the same `HAILER_DATABASE_URL`:
|
|
143
|
+
|
|
144
|
+
```sh
|
|
145
|
+
hailer message-status --id MESSAGE_UUID
|
|
146
|
+
hailer cancel-message --id MESSAGE_UUID --reason "intent withdrawn"
|
|
147
|
+
hailer resolve-message --id MESSAGE_UUID --outcome accepted \
|
|
148
|
+
--provider-message-id VERIFIED_SES_ID --reason "verified provider evidence"
|
|
149
|
+
hailer resolve-message --id MESSAGE_UUID --outcome not-accepted \
|
|
150
|
+
--reason "verified no provider acceptance"
|
|
151
|
+
hailer resolve-message --id MESSAGE_UUID --outcome retry \
|
|
152
|
+
--acknowledge-duplicate-risk --reason "investigated; accept duplicate risk"
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
Cancellation is only possible before dispatch and cannot unsend mail.
|
|
156
|
+
Resolution defaults to the current held-unknown attempt. Use `--attempt-id
|
|
157
|
+
ATTEMPT_UUID` to resolve an older unknown journal entry without overwriting a
|
|
158
|
+
later attempt. Retry is allowed only for the current held intent, with an
|
|
159
|
+
available body, an unexpired deadline, no acceptance latch and explicit
|
|
160
|
+
duplicate-risk acknowledgement. A new attempt never erases the old uncertainty.
|
|
161
|
+
All decisions require an audit reason. After expiry or body purge, sending
|
|
162
|
+
requires a new intent/key.
|
|
163
|
+
|
|
164
|
+
## Quarantine
|
|
165
|
+
|
|
166
|
+
Opt-in, queued pipeline only - a synchronous `:hailer` send has no way to
|
|
167
|
+
hold a message for later review, so a non-ASCII domain or a newly
|
|
168
|
+
registered one makes the synchronous adapter reject outright. The queued
|
|
169
|
+
adapter holds instead:
|
|
170
|
+
|
|
171
|
+
```toml
|
|
172
|
+
[domain_intel]
|
|
173
|
+
provider = "rdap"
|
|
174
|
+
new_domain_threshold_days = 30
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
A recipient domain that is not plain ASCII (a malformed/unencoded IDN, or a
|
|
178
|
+
homograph-spoofing attempt - legitimate IDNs are transmitted as ASCII
|
|
179
|
+
punycode) is always checked, regardless of this block. `[domain_intel]`
|
|
180
|
+
additionally checks the domain's RDAP registration date; newer than the
|
|
181
|
+
threshold holds the message as `quarantined` instead of admitting it as
|
|
182
|
+
`queued`. The whole message is quarantined, not just the flagged recipient -
|
|
183
|
+
this is an operator review gate on intent, not a per-recipient filter like
|
|
184
|
+
suppression or the domain blocklist (`hailer ban-domain`). RDAP's bootstrap
|
|
185
|
+
registry is public; no credential is needed.
|
|
186
|
+
|
|
187
|
+
```sh
|
|
188
|
+
hailer release-quarantine --id MESSAGE_UUID --reason "reviewed, legitimate"
|
|
189
|
+
hailer cancel-message --id MESSAGE_UUID --reason "rejected on review"
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
Releasing resets `next_attempt_at` to now, so the message re-enters the
|
|
193
|
+
normal dispatch queue immediately. `cancel-message` also accepts a
|
|
194
|
+
quarantined message. An unactioned quarantine still expires on the same
|
|
195
|
+
deadline as a normal `queued` message.
|
|
196
|
+
|
|
197
|
+
## Local application verification
|
|
198
|
+
|
|
199
|
+
`test/commonshare_smoke.rb` runs inside CommonShare's test environment with a
|
|
200
|
+
real Mothership bay and loopback SES, never real AWS credentials. Use a fresh
|
|
201
|
+
loopback PostgreSQL database named `hailer_smoke_*` for each queued run.
|
|
202
|
+
The script intentionally reuses one business key to check replay after purge.
|
|
203
|
+
|
|
204
|
+
```sh
|
|
205
|
+
RAILS_ENV=test DISABLE_BOOTSNAP=1 \
|
|
206
|
+
HAILER_SMOKE_QUEUED=1 \
|
|
207
|
+
HAILER_SMOKE_BINARY=/absolute/path/to/current/hailer \
|
|
208
|
+
HAILER_DATABASE_URL=postgres://...@127.0.0.1:5432/hailer_smoke_queued \
|
|
209
|
+
bundle exec rails runner /absolute/path/to/hailer/test/commonshare_smoke.rb
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
An isolated temporary `BUNDLE_GEMFILE` can select the working-tree gem or locally
|
|
213
|
+
installed verification gem without editing CommonShare's Gemfile or lock.
|
|
214
|
+
Set `HAILER_SMOKE_EXPECTED_PLATFORM=arm64-darwin` to assert native-gem selection.
|
|
215
|
+
Omit `HAILER_SMOKE_QUEUED` for the unchanged synchronous contract. The test holds
|
|
216
|
+
the provider response until Rails has its receipt, then checks the journal,
|
|
217
|
+
correlation tags, envelope/attachment, body purge, replay, conflict and feeds.
|
data/hailer.example.toml
CHANGED
|
@@ -1,8 +1,49 @@
|
|
|
1
|
-
#
|
|
2
|
-
state_path = "/var/lib/hailer/state.redb"
|
|
1
|
+
# Set HAILER_DATABASE_URL separately; do not commit credentials.
|
|
3
2
|
threads = 2
|
|
3
|
+
database_pool_size = 5
|
|
4
|
+
shutdown_timeout_seconds = 30
|
|
5
|
+
|
|
6
|
+
# Optional SES event-publishing configuration set. Its events must include mail
|
|
7
|
+
# tags and reach the dedicated SNS/SQS feedback route to resolve unknown sends.
|
|
8
|
+
# Otherwise configure an appropriate default set on the sending identity.
|
|
9
|
+
# ses_configuration_set = "hailer-feedback"
|
|
10
|
+
|
|
11
|
+
# Opt in: initial AWS account/region sync gates send readiness.
|
|
12
|
+
[suppression_sync]
|
|
13
|
+
interval_seconds = 3600
|
|
14
|
+
|
|
15
|
+
# Optional PER-PROCESS budget, not an account-wide SES quota.
|
|
16
|
+
# [rate_limit]
|
|
17
|
+
# recipients_per_second = 10.0
|
|
18
|
+
# burst = 50
|
|
19
|
+
|
|
20
|
+
# In development: durable intake requires explicit logical payload budgets.
|
|
21
|
+
# Requires the durable-dispatch build, not the existing 0.2 release candidates.
|
|
22
|
+
# [queued]
|
|
23
|
+
# max_bytes = 268435456
|
|
24
|
+
# max_messages = 10000
|
|
25
|
+
# unknown_retention_seconds = 604800
|
|
4
26
|
|
|
5
27
|
[feedback]
|
|
6
28
|
# Dedicated queue; subscribe the SNS topic with RawMessageDelivery=false.
|
|
7
29
|
queue_url = "https://sqs.us-east-1.amazonaws.com/123456789012/hailer-feedback"
|
|
8
30
|
topic_arn = "arn:aws:sns:us-east-1:123456789012:mail-feedback"
|
|
31
|
+
|
|
32
|
+
# Opt in: pre-send deliverability check (Emailable today; the provider is a
|
|
33
|
+
# swappable enum, not hardcoded). Set EMAILABLE_API_KEY separately - never
|
|
34
|
+
# here. Verdicts are cached (hailer.address_verifications) so a retry of the
|
|
35
|
+
# same address does not pay for a second lookup.
|
|
36
|
+
# [email_verification]
|
|
37
|
+
# provider = "emailable"
|
|
38
|
+
# reject_states = ["undeliverable"] # risky/unknown fail open by default
|
|
39
|
+
# cache_ttl_seconds = 604800
|
|
40
|
+
# timeout_seconds = 5
|
|
41
|
+
|
|
42
|
+
# Opt in, queued pipeline only: hold a message for operator review
|
|
43
|
+
# (hailer release-quarantine / cancel-message) instead of a hard reject.
|
|
44
|
+
# A non-ASCII recipient domain is always checked regardless of this block;
|
|
45
|
+
# this only adds the RDAP new-domain check. No credential needed - RDAP's
|
|
46
|
+
# bootstrap registry is public.
|
|
47
|
+
# [domain_intel]
|
|
48
|
+
# provider = "rdap"
|
|
49
|
+
# new_domain_threshold_days = 30
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'active_support'
|
|
4
|
+
require 'active_support/current_attributes'
|
|
5
|
+
|
|
6
|
+
module Hailer
|
|
7
|
+
# Scoped by the public ActiveJob callback, restored even on exceptions and
|
|
8
|
+
# nested jobs. No submission key is injected into the rendered MIME.
|
|
9
|
+
class DeliveryContext < ActiveSupport::CurrentAttributes
|
|
10
|
+
attribute :job_id, :ordinal
|
|
11
|
+
|
|
12
|
+
def next_submission_key
|
|
13
|
+
return unless job_id
|
|
14
|
+
|
|
15
|
+
self.ordinal = ordinal.to_i + 1
|
|
16
|
+
"hailer:job:#{job_id}:#{ordinal}"
|
|
17
|
+
end
|
|
18
|
+
end
|
|
19
|
+
end
|
|
@@ -11,6 +11,14 @@ module Hailer
|
|
|
11
11
|
end
|
|
12
12
|
|
|
13
13
|
def deliver!(message)
|
|
14
|
+
Client.new(**settings).deliver(payload_for(message)).tap do |response|
|
|
15
|
+
message.header[:ses_message_id] = response.message_id
|
|
16
|
+
end
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
private
|
|
20
|
+
|
|
21
|
+
def payload_for(message)
|
|
14
22
|
payload = {
|
|
15
23
|
raw_mime: message.to_s,
|
|
16
24
|
to_addresses: Array(explicit_envelope?(message, :to) ? message.smtp_envelope_to : message.to),
|
|
@@ -23,13 +31,9 @@ module Hailer
|
|
|
23
31
|
payload[:from_email_address] = message.smtp_envelope_from
|
|
24
32
|
end
|
|
25
33
|
|
|
26
|
-
|
|
27
|
-
message.header[:ses_message_id] = response.message_id
|
|
28
|
-
end
|
|
34
|
+
payload
|
|
29
35
|
end
|
|
30
36
|
|
|
31
|
-
private
|
|
32
|
-
|
|
33
37
|
# Match the existing SESv2 adapter's distinction between Mail's computed
|
|
34
38
|
# defaults and explicitly assigned SMTP envelope addresses.
|
|
35
39
|
def explicit_envelope?(message, field)
|
data/lib/hailer/errors.rb
CHANGED
|
@@ -19,6 +19,10 @@ module Hailer
|
|
|
19
19
|
end
|
|
20
20
|
|
|
21
21
|
class ValidationError < DeliveryError; end
|
|
22
|
+
class SubmissionConflictError < DeliveryError; end
|
|
22
23
|
class RejectedError < DeliveryError; end
|
|
23
24
|
class UnavailableError < DeliveryError; end
|
|
25
|
+
class RateLimitedError < DeliveryError; end
|
|
26
|
+
# SES might have accepted the message. Automatic retries can duplicate it.
|
|
27
|
+
class AmbiguousDeliveryError < DeliveryError; end
|
|
24
28
|
end
|
|
@@ -0,0 +1,124 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'time'
|
|
4
|
+
|
|
5
|
+
module Hailer
|
|
6
|
+
MessageStatus = Data.define(
|
|
7
|
+
:hailer_message_id, :state, :version, :body_present, :provider_message_id,
|
|
8
|
+
:rfc_message_id, :accepted_at, :expires_at, :attempts, :recipient_outcomes,
|
|
9
|
+
:attempts_truncated, :recipient_outcomes_truncated
|
|
10
|
+
)
|
|
11
|
+
MessageEvent = Data.define(:sequence, :hailer_message_id, :kind, :occurred_at, :details)
|
|
12
|
+
MessageEventPage = Data.define(:events, :next_cursor)
|
|
13
|
+
|
|
14
|
+
# Deliberately separate from the unchanged provider-feedback client/feed.
|
|
15
|
+
class LifecycleClient
|
|
16
|
+
ID = /\A[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\z/
|
|
17
|
+
STATES = %w[queued claimed dispatching retry_waiting held_unknown accepted rejected suppressed cancelled expired].freeze
|
|
18
|
+
MAX_RESPONSE_BYTES = 2 * 1024 * 1024
|
|
19
|
+
|
|
20
|
+
def initialize(**settings)
|
|
21
|
+
@transport = Transport.new(**settings)
|
|
22
|
+
end
|
|
23
|
+
|
|
24
|
+
def status(hailer_message_id)
|
|
25
|
+
unless hailer_message_id.is_a?(String) && ID.match?(hailer_message_id)
|
|
26
|
+
raise ConfigurationError, 'expected a Hailer message UUID'
|
|
27
|
+
end
|
|
28
|
+
document = @transport.get("/internal/mail/messages/#{hailer_message_id}",
|
|
29
|
+
max_response_bytes: MAX_RESPONSE_BYTES)
|
|
30
|
+
receipt = document['receipt']
|
|
31
|
+
unless receipt.is_a?(Hash) && receipt['hailer_message_id'] == hailer_message_id &&
|
|
32
|
+
STATES.include?(document['status']) &&
|
|
33
|
+
document['version'].is_a?(Integer) && document['version'] >= 0 &&
|
|
34
|
+
[true, false].include?(document['body_present']) &&
|
|
35
|
+
optional_text?(document['provider_message_id']) && optional_text?(document['rfc_message_id'])
|
|
36
|
+
raise ProtocolError, 'invalid Hailer message status'
|
|
37
|
+
end
|
|
38
|
+
accepted_at = timestamp(receipt['accepted_at'])
|
|
39
|
+
expires_at = timestamp(receipt['expires_at'])
|
|
40
|
+
raise ProtocolError, 'invalid Hailer message deadline' unless expires_at > accepted_at
|
|
41
|
+
|
|
42
|
+
MessageStatus.new(
|
|
43
|
+
hailer_message_id: hailer_message_id.dup.freeze, state: document['status'].freeze,
|
|
44
|
+
version: document['version'], body_present: document['body_present'],
|
|
45
|
+
provider_message_id: document['provider_message_id']&.freeze,
|
|
46
|
+
rfc_message_id: document['rfc_message_id']&.freeze,
|
|
47
|
+
accepted_at: accepted_at, expires_at: expires_at,
|
|
48
|
+
attempts: status_records(document, 'attempts'),
|
|
49
|
+
recipient_outcomes: status_records(document, 'recipient_outcomes'),
|
|
50
|
+
attempts_truncated: truncation_flag(document, 'attempts_truncated'),
|
|
51
|
+
recipient_outcomes_truncated: truncation_flag(document, 'recipient_outcomes_truncated')
|
|
52
|
+
)
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
def poll(after:, limit: 100)
|
|
56
|
+
unless after.is_a?(Integer) && (0..9_223_372_036_854_775_807).cover?(after) &&
|
|
57
|
+
limit.is_a?(Integer) && (1..100).cover?(limit)
|
|
58
|
+
raise ConfigurationError, 'lifecycle requires a supported nonnegative cursor and limit in 1..100'
|
|
59
|
+
end
|
|
60
|
+
document = @transport.post('/internal/mail/messages/events', { after: after, limit: limit },
|
|
61
|
+
max_response_bytes: MAX_RESPONSE_BYTES)
|
|
62
|
+
entries = document['events']
|
|
63
|
+
unless entries.is_a?(Array) && entries.size <= limit && document['next_cursor'].is_a?(Integer)
|
|
64
|
+
raise ProtocolError, 'invalid Hailer lifecycle page'
|
|
65
|
+
end
|
|
66
|
+
previous = after
|
|
67
|
+
events = entries.map do |entry|
|
|
68
|
+
unless entry.is_a?(Hash) && entry['sequence'].is_a?(Integer) && entry['sequence'] > previous
|
|
69
|
+
raise ProtocolError, 'lifecycle sequences must increase past the requested cursor'
|
|
70
|
+
end
|
|
71
|
+
event = entry['event']
|
|
72
|
+
unless event.is_a?(Hash) && event['hailer_message_id'].is_a?(String) &&
|
|
73
|
+
ID.match?(event['hailer_message_id']) && text?(event['kind']) && event['details'].is_a?(Hash)
|
|
74
|
+
raise ProtocolError, 'invalid Hailer lifecycle event'
|
|
75
|
+
end
|
|
76
|
+
previous = entry['sequence']
|
|
77
|
+
MessageEvent.new(
|
|
78
|
+
sequence: previous, hailer_message_id: event['hailer_message_id'].freeze,
|
|
79
|
+
kind: event['kind'].freeze, occurred_at: timestamp(event['occurred_at']),
|
|
80
|
+
details: freeze_json(event['details'])
|
|
81
|
+
)
|
|
82
|
+
end
|
|
83
|
+
raise ProtocolError, 'lifecycle cursor does not match its page' unless document['next_cursor'] == previous
|
|
84
|
+
|
|
85
|
+
MessageEventPage.new(events: events.freeze, next_cursor: previous)
|
|
86
|
+
end
|
|
87
|
+
|
|
88
|
+
private
|
|
89
|
+
|
|
90
|
+
def text?(value) = value.is_a?(String) && !value.strip.empty?
|
|
91
|
+
def optional_text?(value) = value.nil? || text?(value)
|
|
92
|
+
|
|
93
|
+
def status_records(document, name)
|
|
94
|
+
records = document.fetch(name, [])
|
|
95
|
+
unless records.is_a?(Array) && records.size <= 100 && records.all? { |record| record.is_a?(Hash) }
|
|
96
|
+
raise ProtocolError, 'invalid Hailer status observations'
|
|
97
|
+
end
|
|
98
|
+
freeze_json(records)
|
|
99
|
+
end
|
|
100
|
+
|
|
101
|
+
def truncation_flag(document, name)
|
|
102
|
+
flag = document.fetch(name, false)
|
|
103
|
+
raise ProtocolError, 'invalid Hailer status truncation flag' unless [true, false].include?(flag)
|
|
104
|
+
|
|
105
|
+
flag
|
|
106
|
+
end
|
|
107
|
+
|
|
108
|
+
def timestamp(value)
|
|
109
|
+
raise ProtocolError, 'invalid Hailer lifecycle timestamp' unless text?(value)
|
|
110
|
+
|
|
111
|
+
Time.iso8601(value).freeze
|
|
112
|
+
rescue ArgumentError
|
|
113
|
+
raise ProtocolError, 'invalid Hailer lifecycle timestamp'
|
|
114
|
+
end
|
|
115
|
+
|
|
116
|
+
def freeze_json(value)
|
|
117
|
+
case value
|
|
118
|
+
when Hash then value.each_value { |child| freeze_json(child) }
|
|
119
|
+
when Array then value.each { |child| freeze_json(child) }
|
|
120
|
+
end
|
|
121
|
+
value.freeze
|
|
122
|
+
end
|
|
123
|
+
end
|
|
124
|
+
end
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Hailer
|
|
4
|
+
# Custom application delivery jobs can include this public callback module.
|
|
5
|
+
module DeliveryJobContext
|
|
6
|
+
extend ActiveSupport::Concern
|
|
7
|
+
|
|
8
|
+
included do
|
|
9
|
+
around_perform do |job, perform|
|
|
10
|
+
DeliveryContext.set(job_id: job.job_id, ordinal: 0) { perform.call }
|
|
11
|
+
end
|
|
12
|
+
end
|
|
13
|
+
end
|
|
14
|
+
|
|
15
|
+
class MailDeliveryJob < ActionMailer::MailDeliveryJob
|
|
16
|
+
include DeliveryJobContext
|
|
17
|
+
end
|
|
18
|
+
end
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Hailer
|
|
4
|
+
Acceptance = Data.define(:hailer_message_id)
|
|
5
|
+
|
|
6
|
+
# Acceptance means PostgreSQL owns the message, not that SES accepted it.
|
|
7
|
+
class QueuedClient
|
|
8
|
+
PATH = '/internal/mail/messages'
|
|
9
|
+
|
|
10
|
+
def initialize(intake_attempts: 3, **settings)
|
|
11
|
+
unless intake_attempts.is_a?(Integer) && (1..3).cover?(intake_attempts)
|
|
12
|
+
raise ConfigurationError, 'intake_attempts must be in 1..3'
|
|
13
|
+
end
|
|
14
|
+
@intake_attempts = intake_attempts
|
|
15
|
+
@transport = Transport.new(**settings)
|
|
16
|
+
end
|
|
17
|
+
|
|
18
|
+
def submit(payload, submission_key:, expires_at: nil)
|
|
19
|
+
unless submission_key.is_a?(String) && !submission_key.strip.empty? && submission_key.bytesize <= 256
|
|
20
|
+
raise ConfigurationError, 'submission_key must be nonempty text of at most 256 bytes'
|
|
21
|
+
end
|
|
22
|
+
unless payload.is_a?(Hash) && payload.keys.none? { |key| %w[submission_key expires_at].include?(key.to_s) }
|
|
23
|
+
raise ConfigurationError, 'pass submission_key and expires_at separately from the mail payload'
|
|
24
|
+
end
|
|
25
|
+
command = payload.merge(submission_key: submission_key)
|
|
26
|
+
command[:expires_at] = expires_at unless expires_at.nil?
|
|
27
|
+
# Freeze serialization before the first request: retries must not re-render
|
|
28
|
+
# MIME, regenerate keys, extend deadlines or observe caller mutations.
|
|
29
|
+
body = JSON.generate(command).freeze
|
|
30
|
+
document = submit_frozen(body)
|
|
31
|
+
id = document['hailer_message_id']
|
|
32
|
+
unless id.is_a?(String) && !id.strip.empty?
|
|
33
|
+
raise ProtocolError, 'Hailer accepted the request without a hailer_message_id'
|
|
34
|
+
end
|
|
35
|
+
Acceptance.new(hailer_message_id: id.freeze)
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
private
|
|
39
|
+
|
|
40
|
+
def submit_frozen(body)
|
|
41
|
+
attempts = 0
|
|
42
|
+
begin
|
|
43
|
+
attempts += 1
|
|
44
|
+
@transport.post_serialized(PATH, body, expected_status: 202)
|
|
45
|
+
rescue TransportError, UnavailableError, RateLimitedError
|
|
46
|
+
raise if attempts >= @intake_attempts
|
|
47
|
+
|
|
48
|
+
sleep(0.05 * (2**(attempts - 1)))
|
|
49
|
+
retry
|
|
50
|
+
end
|
|
51
|
+
end
|
|
52
|
+
end
|
|
53
|
+
end
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'securerandom'
|
|
4
|
+
|
|
5
|
+
module Hailer
|
|
6
|
+
class QueuedDeliveryMethod < DeliveryMethod
|
|
7
|
+
def deliver!(message)
|
|
8
|
+
job_key = DeliveryContext.next_submission_key
|
|
9
|
+
key = settings.fetch(:submission_key) { job_key || SecureRandom.uuid }
|
|
10
|
+
options = settings.except(:submission_key, :expires_at)
|
|
11
|
+
QueuedClient.new(**options).submit(
|
|
12
|
+
payload_for(message), submission_key: key, expires_at: settings[:expires_at]
|
|
13
|
+
).tap do |receipt|
|
|
14
|
+
message.header[:hailer_message_id] = receipt.hailer_message_id
|
|
15
|
+
end
|
|
16
|
+
end
|
|
17
|
+
end
|
|
18
|
+
end
|
data/lib/hailer/railtie.rb
CHANGED
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
require 'action_mailer/railtie'
|
|
4
|
+
require_relative 'mail_delivery_job'
|
|
4
5
|
|
|
5
6
|
module Hailer
|
|
6
7
|
class Railtie < ::Rails::Railtie
|
|
@@ -10,9 +11,13 @@ module Hailer
|
|
|
10
11
|
config.hailer = ActiveSupport::OrderedOptions.new
|
|
11
12
|
config.hailer.feedback_handler = nil
|
|
12
13
|
|
|
13
|
-
initializer 'hailer.action_mailer', before: 'action_mailer.set_configs' do
|
|
14
|
+
initializer 'hailer.action_mailer', before: 'action_mailer.set_configs' do |app|
|
|
15
|
+
if app.config.action_mailer.delivery_method == :hailer_queued
|
|
16
|
+
app.config.action_mailer.delivery_job ||= 'Hailer::MailDeliveryJob'
|
|
17
|
+
end
|
|
14
18
|
ActiveSupport.on_load(:action_mailer) do
|
|
15
19
|
add_delivery_method :hailer, Hailer::DeliveryMethod
|
|
20
|
+
add_delivery_method :hailer_queued, Hailer::QueuedDeliveryMethod
|
|
16
21
|
end
|
|
17
22
|
end
|
|
18
23
|
end
|
data/lib/hailer/transport.rb
CHANGED
|
@@ -10,7 +10,7 @@ module Hailer
|
|
|
10
10
|
DEFAULTS = { open_timeout: 5, read_timeout: 30, write_timeout: 30 }.freeze
|
|
11
11
|
MAX_RESPONSE_BYTES = 65_536
|
|
12
12
|
HTTP_ERRORS = {
|
|
13
|
-
400 => ValidationError, 422 => ValidationError,
|
|
13
|
+
400 => ValidationError, 409 => SubmissionConflictError, 422 => ValidationError, 429 => RateLimitedError,
|
|
14
14
|
502 => RejectedError, 503 => UnavailableError
|
|
15
15
|
}.freeze
|
|
16
16
|
|
|
@@ -32,23 +32,37 @@ module Hailer
|
|
|
32
32
|
end
|
|
33
33
|
|
|
34
34
|
def post(path, payload, max_response_bytes: MAX_RESPONSE_BYTES)
|
|
35
|
+
post_serialized(path, JSON.generate(payload), max_response_bytes: max_response_bytes)
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
# Queued intake owns retries and passes the same serialized bytes each time.
|
|
39
|
+
# Synchronous sends continue to require HTTP 200 and never retry.
|
|
40
|
+
def post_serialized(path, body, expected_status: 200, max_response_bytes: MAX_RESPONSE_BYTES)
|
|
35
41
|
request = Net::HTTP::Post.new(path)
|
|
36
42
|
request.content_type = 'application/json'
|
|
43
|
+
request.body = body
|
|
44
|
+
exchange_document(request, expected_status, max_response_bytes)
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
def get(path, max_response_bytes: MAX_RESPONSE_BYTES)
|
|
48
|
+
exchange_document(Net::HTTP::Get.new(path), 200, max_response_bytes)
|
|
49
|
+
end
|
|
50
|
+
|
|
51
|
+
private
|
|
52
|
+
|
|
53
|
+
def exchange_document(request, expected_status, max_response_bytes)
|
|
37
54
|
request['Accept'] = 'application/json'
|
|
38
|
-
request.body = JSON.generate(payload)
|
|
39
55
|
http = Net::HTTP.new(@uri.hostname, @uri.port, nil)
|
|
40
56
|
http.use_ssl = @uri.scheme == 'https'
|
|
41
57
|
http.max_retries = 0
|
|
42
58
|
@timeouts.each { |name, value| http.public_send(:"#{name}=", value) }
|
|
43
59
|
status, body = exchange(http, request, max_response_bytes)
|
|
44
|
-
decode_response(status, body)
|
|
60
|
+
decode_response(status, body, expected_status)
|
|
45
61
|
rescue Timeout::Error, IOError, EOFError, SystemCallError, SocketError,
|
|
46
62
|
OpenSSL::SSL::SSLError, Net::HTTPBadResponse, Net::HTTPHeaderSyntaxError => error
|
|
47
63
|
raise TransportError, "Hailer transport failed (#{error.class}); delivery outcome may be unknown"
|
|
48
64
|
end
|
|
49
65
|
|
|
50
|
-
private
|
|
51
|
-
|
|
52
66
|
def exchange(http, request, limit)
|
|
53
67
|
status = nil
|
|
54
68
|
body = +''
|
|
@@ -65,13 +79,16 @@ module Hailer
|
|
|
65
79
|
[status, body]
|
|
66
80
|
end
|
|
67
81
|
|
|
68
|
-
def decode_response(status, body)
|
|
82
|
+
def decode_response(status, body, expected_status)
|
|
69
83
|
document = JSON.parse(body)
|
|
70
84
|
raise ProtocolError, 'Hailer response must be a JSON object' unless document.is_a?(Hash)
|
|
71
85
|
|
|
72
|
-
unless status ==
|
|
86
|
+
unless status == expected_status
|
|
73
87
|
message = document['error']
|
|
74
88
|
message = "Hailer returned HTTP #{status}" unless message.is_a?(String) && !message.empty?
|
|
89
|
+
if status == 503 && document['code'] == 'outcome_unknown'
|
|
90
|
+
raise AmbiguousDeliveryError.new(message, status: status)
|
|
91
|
+
end
|
|
75
92
|
raise HTTP_ERRORS.fetch(status, DeliveryError).new(message, status: status)
|
|
76
93
|
end
|
|
77
94
|
document
|
data/lib/hailer/version.rb
CHANGED
data/lib/hailer.rb
CHANGED
|
@@ -5,6 +5,10 @@ require_relative 'hailer/errors'
|
|
|
5
5
|
require_relative 'hailer/transport'
|
|
6
6
|
require_relative 'hailer/client'
|
|
7
7
|
require_relative 'hailer/delivery_method'
|
|
8
|
+
require_relative 'hailer/queued_client'
|
|
9
|
+
require_relative 'hailer/delivery_context'
|
|
10
|
+
require_relative 'hailer/queued_delivery_method'
|
|
11
|
+
require_relative 'hailer/lifecycle_client'
|
|
8
12
|
require_relative 'hailer/feedback_client'
|
|
9
13
|
require_relative 'hailer/feedback_consumer'
|
|
10
14
|
require_relative 'hailer/railtie'
|