hailer 0.1.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +24 -1
- data/README.md +299 -32
- data/docs/durable-delivery.md +217 -0
- data/hailer.example.toml +43 -2
- data/lib/hailer/delivery_context.rb +19 -0
- data/lib/hailer/delivery_method.rb +9 -5
- data/lib/hailer/errors.rb +4 -0
- data/lib/hailer/lifecycle_client.rb +124 -0
- data/lib/hailer/mail_delivery_job.rb +18 -0
- data/lib/hailer/queued_client.rb +53 -0
- data/lib/hailer/queued_delivery_method.rb +18 -0
- data/lib/hailer/railtie.rb +6 -1
- data/lib/hailer/transport.rb +24 -7
- data/lib/hailer/version.rb +1 -1
- data/lib/hailer.rb +4 -0
- metadata +7 -3
- data/docs/delivery-contract.md +0 -59
- data/docs/feedback.md +0 -160
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 141f02c631f9a0c14ee84232561c516e40a33f9e0683ae31112c293a98011174
|
|
4
|
+
data.tar.gz: 3881311e389ab55c5b4d79c847f8c4f6da945666379bdfac998bdb534d7b3566
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 1ed05d6d91f27d9054f19267d7a620cbb4bee68ab218dafc7f3a284e8403363414faff7c2cb83228f6987a326cc3c457d825eb0f5898634dfaefa60d1ee805fc
|
|
7
|
+
data.tar.gz: b485850221aac71b4634e06c55b12e0d605c9de789e9390f1227133b278921dd807f1bf575602829d93aa0e708df14b91149ad4679c9bbce01ccd0c95cc0e92a
|
data/CHANGELOG.md
CHANGED
|
@@ -1,6 +1,29 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
-
##
|
|
3
|
+
## Working tree — durable delivery (unreleased)
|
|
4
|
+
|
|
5
|
+
- Opt-in `:hailer_queued` Rails adapter with stable job keys and frozen intake retries.
|
|
6
|
+
- PostgreSQL ownership receipts, immutable submission replay/conflict and explicit spool budgets.
|
|
7
|
+
- Supervised fenced dispatch, journaled attempts, safe retries and held unknown outcomes.
|
|
8
|
+
- Tagged SES feedback correlation, recipient facts, audited cancellation and resolution.
|
|
9
|
+
- Deadline/body retention, independent lifecycle feed and intake/dispatcher readiness.
|
|
10
|
+
- Native Rust future traits without `async-trait`; throttle-machines 0.3.1.
|
|
11
|
+
- Current-source CommonShare queued smoke with loopback SES; no production cutover.
|
|
12
|
+
- New Linux binary builds target Debian 13 Trixie (glibc 2.41), with locked GNU Ruby dependency variants.
|
|
13
|
+
- Opt-in pre-send address verification (Emailable first; swappable via `Verifier` enum), `[email_verification]` in `hailer.toml`, cached verdicts in `hailer.address_verifications`. Fails open by default - a verifier outage or `unknown`/`risky` result never blocks a send unless explicitly configured into `reject_states`.
|
|
14
|
+
- Domain/pattern blocklist (`hailer.domain_blocks`): exact domains or `*.`-suffix globs, always active. A small hand-curated disposable-provider list ships with the repo (source `curated`, only changes by upgrading); operators add their own via `hailer ban-domain`/`unban-domain` (source `operator`). Checked before suppression and verification - the cheapest, broadest filter runs first.
|
|
15
|
+
- New `quarantined` message status, queued pipeline only: a non-ASCII recipient domain, or (opt-in `[domain_intel]`, RDAP-backed) a newly registered one, holds the whole message for operator review (`hailer release-quarantine`/`cancel-message`) instead of a hard reject. Excluded from the dispatch index like `held_unknown`; expires on the normal deadline like `queued`.
|
|
16
|
+
|
|
17
|
+
## 0.2.0 (unreleased)
|
|
18
|
+
|
|
19
|
+
- PostgreSQL-only live state, with atomic feedback and commit-safe cursors.
|
|
20
|
+
- Resumable, read-only AWS suppression reconciliation scoped by account/region.
|
|
21
|
+
- Pure behavior admission circuit, per-process throttle, and SES circuit breaker.
|
|
22
|
+
- Single-attempt sends, explicit ambiguous outcomes, and typed rate-limit errors.
|
|
23
|
+
- Private health/readiness sensors, supervised workers, and bounded graceful drain.
|
|
24
|
+
- Explicit read-only migration of stopped 0.1 redb state into an empty database.
|
|
25
|
+
|
|
26
|
+
## 0.1.0
|
|
4
27
|
|
|
5
28
|
- Rails 8.1+/Ruby 4.0+ ActionMailer delivery with automatic defaults.
|
|
6
29
|
- Typed provider-neutral feedback client and Rails callback consumer.
|
data/README.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
Rails writes the email. Hailer deals with sending it.
|
|
4
4
|
|
|
5
|
-
This repository
|
|
5
|
+
This repository prepares two things at version **0.2.0** (not published yet):
|
|
6
6
|
|
|
7
7
|
- **`hailer` gem** — an ActionMailer adapter, feedback client, and executable
|
|
8
8
|
launcher. Platform gems bundle the Rust binary. No Ruby AWS SDK or native extension.
|
|
@@ -13,6 +13,12 @@ This repository ships two things at version **0.1.0**:
|
|
|
13
13
|
ActionMailer → hailer gem → private Mothership route → hailer binary → SES
|
|
14
14
|
```
|
|
15
15
|
|
|
16
|
+
The working tree also implements an **opt-in durable queued adapter**. See
|
|
17
|
+
[durable delivery](docs/durable-delivery.md) for ownership, recovery, routing
|
|
18
|
+
and operator controls. It is not included in the preserved 0.2 release
|
|
19
|
+
candidates. Current-source macOS, ARM64 Linux and x86_64 Linux verification
|
|
20
|
+
gates pass locally; this work has not been published or deployed.
|
|
21
|
+
|
|
16
22
|
## Rails
|
|
17
23
|
|
|
18
24
|
Requires Ruby 4.0+ and Rails 8.1 or newer within Rails 8.x.
|
|
@@ -78,7 +84,7 @@ name = "hailer"
|
|
|
78
84
|
command = "hailer"
|
|
79
85
|
workers = 0
|
|
80
86
|
threads = 2
|
|
81
|
-
routes = [{ bind = "mail", pattern = "^/internal/mail/send$" }]
|
|
87
|
+
routes = [{ bind = "mail", pattern = "^/internal/mail/(send|events|health|ready)$" }]
|
|
82
88
|
```
|
|
83
89
|
|
|
84
90
|
Mothership exports `MS_PID`, `MS_SHIP`, and `MS_SOCKET_DIR`. Hailer binds its
|
|
@@ -90,6 +96,31 @@ Give the **service**, not the Rails adapter, AWS region and credentials through
|
|
|
90
96
|
the standard AWS credential chain. Its identity needs `ses:SendEmail`, and the
|
|
91
97
|
SES account/identities must be configured for sending.
|
|
92
98
|
|
|
99
|
+
### PostgreSQL
|
|
100
|
+
|
|
101
|
+
PostgreSQL is the **only live backend** in 0.2. Hailer owns a `hailer` schema,
|
|
102
|
+
not Rails tables. Use a separate database/role where practical:
|
|
103
|
+
|
|
104
|
+
```sh
|
|
105
|
+
export HAILER_DATABASE_URL='postgres://hailer:password@db/hailer?sslmode=verify-full'
|
|
106
|
+
hailer migrate
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
Supply the database CA through the normal PostgreSQL connection options.
|
|
110
|
+
Use certificate verification in deployment; a private network is not TLS.
|
|
111
|
+
Keep the URL in a secret manager, not TOML or a committed initializer.
|
|
112
|
+
Startup also applies embedded, versioned migrations. The bootstrap identity
|
|
113
|
+
needs database `CREATE` and ownership of its schema/tables. Runtime startup
|
|
114
|
+
currently runs that same bootstrap/migrator, so provision the service role for
|
|
115
|
+
it too; a separate DML-only runtime mode is not implemented.
|
|
116
|
+
The runtime never needs Rails schema ownership. Back up the database with your
|
|
117
|
+
normal PostgreSQL backup/restore tooling.
|
|
118
|
+
|
|
119
|
+
Configuration defaults: pool size 5 (range 1–64), shutdown timeout 30 seconds
|
|
120
|
+
(range 1–300). Every replica shares PostgreSQL; each has its own pool and optional
|
|
121
|
+
rate budget. Set your supervisor/container stop grace longer than Hailer's
|
|
122
|
+
shutdown timeout. PostgreSQL 18 is exercised locally and in CI.
|
|
123
|
+
|
|
93
124
|
## Bounces, complaints, and suppressions
|
|
94
125
|
|
|
95
126
|
```text
|
|
@@ -97,7 +128,7 @@ SES → SNS → dedicated SQS queue → Hailer → durable events + suppressions
|
|
|
97
128
|
```
|
|
98
129
|
|
|
99
130
|
Hailer atomically commits each event, deduplication key, and suppression change
|
|
100
|
-
to
|
|
131
|
+
to PostgreSQL **before** acknowledging SQS. Failed acknowledgements
|
|
101
132
|
can be retried without duplicating stored events. Invalid/untrusted messages
|
|
102
133
|
are left for SQS's dead-letter policy.
|
|
103
134
|
|
|
@@ -113,14 +144,16 @@ are left for SQS's dead-letter policy.
|
|
|
113
144
|
Copy [hailer.example.toml](hailer.example.toml), set the real queue/topic, and
|
|
114
145
|
pass `--config /path/to/hailer.toml` or `HAILER_CONFIG`. Without `[feedback]`,
|
|
115
146
|
the consumer is disabled, but previously stored suppressions are still enforced.
|
|
116
|
-
`
|
|
117
|
-
|
|
147
|
+
`HAILER_DATABASE_URL` is required, even with feedback disabled.
|
|
148
|
+
The old `state_path`, `--state-path`, and `HAILER_STATE_PATH` are not supported.
|
|
149
|
+
Replicas may consume the same dedicated queue when they share the same Hailer
|
|
150
|
+
database. Commits serialize cursor allocation so a reader cannot skip an earlier
|
|
151
|
+
transaction that has not committed yet.
|
|
118
152
|
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
no automatic retention/pruning.
|
|
153
|
+
Event history and deduplication have **no automatic retention/pruning**.
|
|
154
|
+
Do not delete events casually: event IDs also carry the SQS deduplication
|
|
155
|
+
contract, and Rails consumers may still need their cursors. Plan retention
|
|
156
|
+
against consumer lag and redelivery windows before adding a pruning policy.
|
|
124
157
|
|
|
125
158
|
Hailer also exposes a provider-neutral, nondestructive event feed:
|
|
126
159
|
|
|
@@ -157,12 +190,159 @@ updates in one application database transaction. Retried pages require
|
|
|
157
190
|
idempotent handlers; no hidden poller, scheduling, or checkpoint database is
|
|
158
191
|
started by the gem. Reading does not acknowledge or erase events.
|
|
159
192
|
|
|
160
|
-
|
|
193
|
+
The SQS policy must allow only the intended SNS topic, using `aws:SourceArn`
|
|
194
|
+
and account restrictions. Subscribe with `RawMessageDelivery=false`, restrict
|
|
195
|
+
direct `sqs:SendMessage`, set a dead-letter queue, and monitor its depth.
|
|
196
|
+
The SNS topic allowlist in TOML is defense in depth, not signature verification.
|
|
197
|
+
Hailer needs `sqs:ReceiveMessage` and `sqs:DeleteMessage` on that queue, plus
|
|
198
|
+
applicable KMS decrypt permissions if customer-managed encryption is used.
|
|
199
|
+
|
|
200
|
+
### Import AWS suppressions
|
|
201
|
+
|
|
202
|
+
```sh
|
|
203
|
+
hailer sync-suppressions
|
|
204
|
+
# If AWS expires a saved pagination token:
|
|
205
|
+
hailer sync-suppressions --restart
|
|
206
|
+
# Existing application export, JSON array:
|
|
207
|
+
hailer import-suppressions --file suppressions.json
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
The service identity needs `ses:ListSuppressedDestinations`; STS
|
|
211
|
+
`GetCallerIdentity` identifies the account (AWS does not require an explicit
|
|
212
|
+
allow for that operation). The configured region is part of the source identity.
|
|
213
|
+
These commands **read AWS only**; they never mutate the account suppression list.
|
|
214
|
+
Missing AWS entries do not unsuppress local entries, and a bounce never
|
|
215
|
+
downgrades a complaint. Imports do not fabricate delivery-history events.
|
|
216
|
+
Account-level AWS data does not include every possible global SES suppression.
|
|
217
|
+
|
|
218
|
+
Enable `[suppression_sync]` as in the example to reconcile on startup and hourly.
|
|
219
|
+
The first complete sync gates send readiness. Later errors mark health degraded,
|
|
220
|
+
retain committed pages, and retry safe reads with bounded backoff. A token is
|
|
221
|
+
stored with its page in one transaction. Replicas coordinate a source-scoped
|
|
222
|
+
advisory lock; they cannot advance the same checkpoint concurrently.
|
|
223
|
+
An expired token deliberately requires the operator's `--restart`. Omitting
|
|
224
|
+
the section disables scheduled sync, not existing suppression enforcement.
|
|
225
|
+
|
|
226
|
+
### Control plane
|
|
227
|
+
|
|
228
|
+
The behavior circuit evaluates **pure admission decisions**: input validity,
|
|
229
|
+
service/storage readiness, suppression, and local rate admission. PostgreSQL,
|
|
230
|
+
AWS calls, clocks, and token reservations remain owned by the async host.
|
|
231
|
+
The SES circuit breaker counts transient/ambiguous failures, not bad addresses
|
|
232
|
+
or permanent request rejections. An optional `[rate_limit]` budgets envelope
|
|
233
|
+
recipients **per process**, not per account; it does not discover SES quotas.
|
|
234
|
+
|
|
235
|
+
Private `GET /internal/mail/health` returns sanitized component states and
|
|
236
|
+
counters. `GET /internal/mail/ready` returns 503 during initial sync or storage
|
|
237
|
+
failure, otherwise 200. A provider circuit can be degraded while readiness
|
|
238
|
+
remains true so protected half-open probes can recover it. No credentials,
|
|
239
|
+
recipient addresses, message bodies, or pagination tokens appear in these sensors.
|
|
240
|
+
They are operational signals, not a guarantee SES will accept a message.
|
|
241
|
+
|
|
242
|
+
TERM/ctrl-c stops admission and SQS receives, then drains owned requests and
|
|
243
|
+
the current feedback commit/ack within the configured deadline. Idle links
|
|
244
|
+
close immediately. Deadline expiry aborts remaining tasks and exits nonzero;
|
|
245
|
+
an interrupted send can have an unknown outcome.
|
|
161
246
|
|
|
162
247
|
**Security:** the route currently has no application-level authentication.
|
|
163
248
|
`/internal/` is a name, not an access control. Keep the bind private. The client
|
|
164
249
|
does not follow redirects or use environment HTTP proxies.
|
|
165
250
|
|
|
251
|
+
## Impossible addresses
|
|
252
|
+
|
|
253
|
+
Always on, no configuration: `toto@example.com` is syntactically a real
|
|
254
|
+
address, but RFC 2606 reserves `example.com`/`.net`/`.org` and the
|
|
255
|
+
`test`/`example`/`invalid`/`localhost` TLDs for documentation - no real
|
|
256
|
+
mailbox ever lives there. Rejected at the same boundary as a malformed
|
|
257
|
+
address (`SendRequest::validate`), before any guard in the chain runs, for
|
|
258
|
+
both the synchronous and queued adapters. Not a policy choice like the
|
|
259
|
+
domain blocklist below - there is nothing to configure or an operator could
|
|
260
|
+
turn off.
|
|
261
|
+
|
|
262
|
+
## Domain blocklist
|
|
263
|
+
|
|
264
|
+
```text
|
|
265
|
+
SendRequest -> impossible-address check -> domain blocklist (free, local) -> suppression (free, local) -> verification (paid, external) -> SES
|
|
266
|
+
```
|
|
267
|
+
|
|
268
|
+
Always active - an empty `hailer.domain_blocks` table is a no-op, there is
|
|
269
|
+
no toggle. A pattern is an exact domain (`guerrillamail.com`) or a
|
|
270
|
+
`*.`-prefixed suffix glob (`*.kp` matches that domain and any subdomain).
|
|
271
|
+
A small hand-curated list of known disposable-mail providers ships with the
|
|
272
|
+
repo (`source = 'curated'`, migration `05_domain_blocks.sql`); it only grows
|
|
273
|
+
by upgrading Hailer, never fetched from a third party at runtime. Operators
|
|
274
|
+
add their own:
|
|
275
|
+
|
|
276
|
+
```sh
|
|
277
|
+
hailer ban-domain --pattern "*.kp" --reason "sanctioned TLD"
|
|
278
|
+
hailer unban-domain --pattern "*.kp"
|
|
279
|
+
```
|
|
280
|
+
|
|
281
|
+
`unban-domain` only removes an `operator`-sourced pattern - a curated one
|
|
282
|
+
cannot be removed this way, by design.
|
|
283
|
+
|
|
284
|
+
## Address verification
|
|
285
|
+
|
|
286
|
+
```text
|
|
287
|
+
SendRequest -> suppression check (cheap, local) -> address verification (paid, external) -> SES
|
|
288
|
+
```
|
|
289
|
+
|
|
290
|
+
Opt-in pre-send deliverability check, off by default. Enable with
|
|
291
|
+
`[email_verification]` in `hailer.toml`:
|
|
292
|
+
|
|
293
|
+
```toml
|
|
294
|
+
[email_verification]
|
|
295
|
+
provider = "emailable"
|
|
296
|
+
reject_states = ["undeliverable"] # risky/unknown fail open unless added here
|
|
297
|
+
cache_ttl_seconds = 604800
|
|
298
|
+
timeout_seconds = 5
|
|
299
|
+
```
|
|
300
|
+
|
|
301
|
+
Set `EMAILABLE_API_KEY` in the process environment, never in `hailer.toml` -
|
|
302
|
+
the same rule as `HAILER_DATABASE_URL` and AWS credentials. Suppression
|
|
303
|
+
filters first since it is a free local lookup; verification only runs for
|
|
304
|
+
recipients that survive it, so a suppressed address never costs a paid
|
|
305
|
+
lookup. Verdicts are cached in `hailer.address_verifications` (a week by
|
|
306
|
+
default) - a retried send against the same address reuses the cached verdict
|
|
307
|
+
instead of paying for another check.
|
|
308
|
+
|
|
309
|
+
A verifier outage, timeout, or an `unknown`/`risky` result all fail open: this
|
|
310
|
+
middleware is never a new reason a real send stops working unless a
|
|
311
|
+
deployment explicitly adds those states to `reject_states`. Adding another
|
|
312
|
+
provider (ZeroBounce, NeverBounce, ...) means a new `Verifier` enum variant in
|
|
313
|
+
`src/verify.rs`; nothing else in the send path changes.
|
|
314
|
+
|
|
315
|
+
### Bulk pre-screening (batch)
|
|
316
|
+
|
|
317
|
+
A separate concern from the real-time guard above: pre-verify a whole list
|
|
318
|
+
*before* ever trying to send to it, not a per-send check. Emailable's batch
|
|
319
|
+
API is async (submit, then poll) and only returns per-address results inline
|
|
320
|
+
for batches up to 1,000 - larger ones get a ZIP/CSV download link this client
|
|
321
|
+
deliberately does not fetch or parse, to avoid a zip/csv dependency:
|
|
322
|
+
|
|
323
|
+
```sh
|
|
324
|
+
hailer verify-batch --file addresses.txt # one address per line, max 1000
|
|
325
|
+
hailer batch-status --id BATCH_ID # poll; caches each verdict once complete
|
|
326
|
+
```
|
|
327
|
+
|
|
328
|
+
`batch-status` writes every returned verdict into `hailer.address_verifications`
|
|
329
|
+
(the same cache the real-time guard reads), so a later send to an
|
|
330
|
+
already-screened address reuses the cached verdict instead of paying for
|
|
331
|
+
another single-address check. A batch over 1,000 addresses is reported with
|
|
332
|
+
its `download_file` URL but is not cached - split into smaller batches
|
|
333
|
+
instead.
|
|
334
|
+
|
|
335
|
+
`verify-batch` never submits an address that does not need it: exact
|
|
336
|
+
duplicate lines (by the same normalized key the cache itself is keyed by -
|
|
337
|
+
`USER+alias@x` and `user@x` are the same address), and anything already in
|
|
338
|
+
`hailer.address_verifications` with an unexpired verdict, are dropped before
|
|
339
|
+
the batch is ever submitted - never spending a credit on them. Every
|
|
340
|
+
exclusion is counted and reported (`invalid_lines`, `duplicate_lines`,
|
|
341
|
+
`already_cached`), never silently dropped. `hailer emailable-account` checks
|
|
342
|
+
the Emailable account's remaining credits directly; `submit_batch` itself
|
|
343
|
+
also checks this before submitting and refuses (without ever reaching the
|
|
344
|
+
server) if the account has none left or fewer than the batch needs.
|
|
345
|
+
|
|
166
346
|
## Failure semantics
|
|
167
347
|
|
|
168
348
|
An HTTP 200 means **SES accepted the message**, not that the recipient received
|
|
@@ -172,28 +352,47 @@ it. A successful response must include a nonblank `message_id`.
|
|
|
172
352
|
| --- | --- |
|
|
173
353
|
| HTTP 400 / 422 | `Hailer::ValidationError` |
|
|
174
354
|
| HTTP 502 | `Hailer::RejectedError` |
|
|
355
|
+
| HTTP 429 | `Hailer::RateLimitedError` |
|
|
356
|
+
| HTTP 503, `code: "outcome_unknown"` | `Hailer::AmbiguousDeliveryError` |
|
|
175
357
|
| HTTP 503 | `Hailer::UnavailableError` |
|
|
176
358
|
| Other non-200 status | `Hailer::DeliveryError` |
|
|
177
359
|
| Invalid/oversized response, missing message ID | `Hailer::ProtocolError` |
|
|
178
360
|
| Connection, TLS, or timeout failure | `Hailer::TransportError` |
|
|
179
361
|
| Missing/invalid endpoint or timeout settings | `Hailer::ConfigurationError` |
|
|
180
362
|
|
|
181
|
-
The
|
|
182
|
-
message; retrying may duplicate it.
|
|
183
|
-
|
|
363
|
+
The synchronous adapter does not retry. A lost response can happen **after** SES
|
|
364
|
+
accepted the message; retrying may duplicate it. This route does not use the
|
|
365
|
+
opt-in outbound spool or its submission-key deduplication.
|
|
184
366
|
Rails jobs can also retry, so configure them deliberately.
|
|
185
|
-
The
|
|
186
|
-
|
|
367
|
+
The SES SDK makes **one send attempt**. AWS throttling/transient failures return
|
|
368
|
+
503, permanent service rejections 502, and disconnects/timeouts/missing message
|
|
369
|
+
IDs carry the explicit unknown-outcome code. None of these responses authorizes
|
|
370
|
+
an automatic retry of an ambiguous send.
|
|
187
371
|
|
|
188
372
|
## Status and migration
|
|
189
373
|
|
|
190
374
|
The delivery contract and Hailer-side feedback processing are implemented. CommonShare's
|
|
191
375
|
existing sender is the compatibility reference, not a runtime dependency.
|
|
192
|
-
|
|
376
|
+
|
|
377
|
+
### Upgrade from 0.1 redb
|
|
378
|
+
|
|
379
|
+
1. Stop all old Hailer processes cleanly. Keep the source file and a backup.
|
|
380
|
+
2. Provision an **empty** Hailer PostgreSQL database and set its URL.
|
|
381
|
+
3. Run `hailer migrate-redb --file /private/path/state.redb`.
|
|
382
|
+
4. Check reported counts, event cursors, and suppression enforcement before
|
|
383
|
+
starting 0.2. Keep the source backup; do not restart old writers after cutover.
|
|
384
|
+
|
|
385
|
+
Migration opens redb read-only, checks event/dedup mappings and suppression keys,
|
|
386
|
+
and copies bounded batches in **one destination transaction**. It preserves
|
|
387
|
+
event IDs, sequence values, suppression metadata, and next-cursor continuity.
|
|
388
|
+
It rejects missing/corrupt sources and nonempty targets without wiping anything.
|
|
389
|
+
An unclean redb file may require recovery with the old version against a backup
|
|
390
|
+
before migration. There is no implicit fallback and no dual-write period.
|
|
193
391
|
|
|
194
392
|
Before switching an existing app:
|
|
195
393
|
|
|
196
|
-
1.
|
|
394
|
+
1. Provision PostgreSQL, run Hailer on a private Mothership bind, check readiness,
|
|
395
|
+
and verify real SES delivery using a deliberate test.
|
|
197
396
|
2. Add the gem, set `HAILER_URL`, and remove the app's old delivery-method
|
|
198
397
|
override so Hailer's default takes effect.
|
|
199
398
|
3. Subscribe a dedicated SQS queue before exporting and importing existing
|
|
@@ -202,22 +401,26 @@ Before switching an existing app:
|
|
|
202
401
|
and any provider-neutral app history consumer have been verified.
|
|
203
402
|
5. Only then remove the old sender and AWS-specific feedback handling.
|
|
204
403
|
|
|
205
|
-
CommonShare's local integration is tested with its real application,
|
|
206
|
-
Mothership,
|
|
404
|
+
CommonShare's local 0.2 integration is tested with its real application,
|
|
405
|
+
test database, Mothership, the optimized binary from the macOS gem, its installed 0.1 Ruby
|
|
406
|
+
client, and a loopback SES fixture. Its existing AWS
|
|
207
407
|
receivers remain until the real subscription and suppression cutover is verified.
|
|
208
408
|
No deployed AWS routing has been changed.
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
|
|
409
|
+
The current-source queued adapter and dispatcher also pass a CommonShare smoke
|
|
410
|
+
using an isolated Bundler Gemfile, real Mothership and loopback SES. No app
|
|
411
|
+
Gemfile or production configuration was changed. Its legacy history observer
|
|
412
|
+
still needs queued-ownership semantics before opt-in deployment.
|
|
413
|
+
Validation/behavior plugins and distributed account-wide quotas remain later
|
|
414
|
+
work. No plugin API or provider exactly-once delivery is promised.
|
|
212
415
|
|
|
213
416
|
## Development
|
|
214
417
|
|
|
215
418
|
```sh
|
|
216
419
|
bundle install
|
|
217
420
|
bundle exec rake
|
|
218
|
-
cargo test --locked
|
|
421
|
+
HAILER_TEST_DATABASE_URL=postgres://test_user:password@127.0.0.1/hailer_test cargo test --locked
|
|
219
422
|
cargo fmt --all -- --check
|
|
220
|
-
gem build hailer.gemspec --output /tmp/hailer-0.
|
|
423
|
+
gem build hailer.gemspec --output /tmp/hailer-0.2.0.gem
|
|
221
424
|
bundle exec rake native:package
|
|
222
425
|
```
|
|
223
426
|
|
|
@@ -225,17 +428,81 @@ The Ruby suite exercises real local HTTP exchanges and a booted Rails app.
|
|
|
225
428
|
The Rust suite checks the same JSON fixture and the docking exchange with a
|
|
226
429
|
fake provider. Feedback tests cover redelivery, acknowledgement ordering,
|
|
227
430
|
atomic rollback, restart persistence, imports, and suppression enforcement.
|
|
228
|
-
|
|
431
|
+
Rust tests require an isolated PostgreSQL server and a test role with `CREATEDB`;
|
|
432
|
+
each test creates and drops its own database. Never point this variable at a
|
|
433
|
+
customer or Rails production database. Neither suite sends real mail or requires
|
|
434
|
+
AWS credentials.
|
|
435
|
+
|
|
436
|
+
For the opt-in CommonShare compatibility smoke, create a disposable Hailer
|
|
437
|
+
database, then run from that application's checkout:
|
|
438
|
+
|
|
439
|
+
```sh
|
|
440
|
+
RAILS_ENV=test HAILER_SMOKE_BINARY=/path/to/hailer/target/debug/hailer \
|
|
441
|
+
HAILER_DATABASE_URL=postgres://test_user:password@127.0.0.1/hailer_smoke \
|
|
442
|
+
bundle exec rails runner /path/to/hailer/test/commonshare_smoke.rb
|
|
443
|
+
```
|
|
444
|
+
|
|
445
|
+
It refuses a non-test app database, points the AWS SDK at a loopback fixture,
|
|
446
|
+
and rolls back application-history changes. It does not change the app's
|
|
447
|
+
dependency lockfile or send external mail.
|
|
229
448
|
|
|
230
449
|
`native:package` builds a locked release and writes
|
|
231
|
-
`pkg/hailer-0.
|
|
450
|
+
`pkg/hailer-0.2.0-<platform>.gem`. Each platform gem contains exactly its host
|
|
232
451
|
binary and a version/platform/SHA-256 manifest. The launcher verifies that
|
|
233
452
|
manifest before `exec`, preserving arguments, exit status, and signal handling.
|
|
234
453
|
`HAILER_BINARY` is an explicit, version-checked source-build override; there is
|
|
235
454
|
no download or PATH fallback.
|
|
236
455
|
|
|
237
|
-
Build on each supported release host before publishing its artifact.
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
456
|
+
Build on each supported release host before publishing its artifact. Nothing
|
|
457
|
+
in the packaging task publishes a gem.
|
|
458
|
+
|
|
459
|
+
## Linux releases and RubyGems
|
|
460
|
+
|
|
461
|
+
The **Linux binary gems** GitHub Actions workflow builds `x86_64-linux-gnu`
|
|
462
|
+
and `aarch64-linux-gnu` natively against Debian 13 Trixie (glibc 2.41).
|
|
463
|
+
New Trixie-built artifacts target glibc 2.41 or newer, not Bookworm or
|
|
464
|
+
Alpine/musl. The preserved older 0.2 candidates retain their original baseline.
|
|
465
|
+
Local OrbStack verification uses `test/linux/Dockerfile` for each architecture
|
|
466
|
+
and a separate `ruby:4.0-slim-trixie` runtime.
|
|
467
|
+
Use a copied checkout in separate per-architecture Docker work volumes; do not
|
|
468
|
+
reuse the macOS `target/` or Ruby bundle. Run the locked Ruby tests, release Rust
|
|
469
|
+
tests and `native:package` with an isolated PostgreSQL fixture, then install each
|
|
470
|
+
gem in its slim runtime and exercise the launcher, migrations and import CLI.
|
|
471
|
+
When transferring a macOS snapshot, use `COPYFILE_DISABLE=1 tar --no-xattrs`
|
|
472
|
+
so AppleDouble `._*` metadata cannot appear as spurious SQL migrations.
|
|
473
|
+
|
|
474
|
+
```sh
|
|
475
|
+
gh workflow run linux-gems.yml --repo seuros/hailer -f tag=v0.2.0
|
|
476
|
+
```
|
|
477
|
+
|
|
478
|
+
The workflow checks out the requested release tag, runs the Ruby and Rust
|
|
479
|
+
suites against an isolated PostgreSQL 18 service, and installs each binary gem
|
|
480
|
+
in a separate slim Debian container. Pushes/PRs also verify the branch without
|
|
481
|
+
creating a release or publishing anything; only explicit workflow dispatch
|
|
482
|
+
can attach assets to an existing release.
|
|
483
|
+
It verifies the installed platform, bundled-binary manifest, executable version,
|
|
484
|
+
and PostgreSQL suppression import before attaching either Linux gem to the
|
|
485
|
+
existing release. Existing gem assets are never overwritten; `SHA256SUMS` is
|
|
486
|
+
updated to cover all release gems.
|
|
487
|
+
|
|
488
|
+
Publishing a GitHub release does **not** publish to RubyGems. Like Shuttlebay,
|
|
489
|
+
release assets can be fetched and published with Rake:
|
|
490
|
+
|
|
491
|
+
```sh
|
|
492
|
+
bundle exec rake gems:fetch
|
|
493
|
+
bundle exec rake gems:push_all
|
|
494
|
+
# Or select an explicit release:
|
|
495
|
+
bundle exec rake 'gems:push_all[v0.2.0]'
|
|
496
|
+
```
|
|
497
|
+
|
|
498
|
+
`gems:push_all` fetches fresh assets, checks every archive's SHA-256,
|
|
499
|
+
name/version/platform, and publishes only that verified set. It skips
|
|
500
|
+
already-published platforms only when RubyGems reports the identical archive
|
|
501
|
+
hash, so an interrupted publication can be resumed without replacing artifacts.
|
|
502
|
+
`gems:push` is an alias. No release task rebuilds or silently republishes gems.
|
|
503
|
+
|
|
504
|
+
RubyGems may require MFA or a suitably scoped API key. Keep credentials out of
|
|
505
|
+
the repository. Publishing makes the packaged Ruby code and binaries public,
|
|
506
|
+
even when the GitHub repository is private. Once published, Bundler selects
|
|
507
|
+
the matching native gem; do not set `force_ruby_platform` in deployment if
|
|
508
|
+
you want the bundled executable.
|