cloudflare-email 0.1.0 → 0.2.0

data/docs/troubleshooting.md

@@ -0,0 +1,103 @@
+# When email does not behave as expected
+
+Start by identifying which step failed: sending, incoming storage, mailbox
+processing, or delivery tracking. Those are separate pieces. Use test addresses
+you control, and avoid pasting tokens, raw MIME or private provider responses
+into public logs or issues.
+
+## Sending
+
+| What you see | What to check next |
+| --- | --- |
+| New generator or API is missing | Check `Gemfile.lock`. The new features require the [0.2 commit](getting-started.md#install-the-current-code); published 0.1.0 lacks them. |
+| Credentials appear to be ignored | Nonempty Rails credentials override environment variables. Check the Rails environment and restart the app after changes. |
+| Authentication or domain error | Run `bin/rails cloudflare:email:doctor`; check account ID, token permissions and sending-domain verification. |
+| `doctor` reports limited read access | A send token may lack diagnostic read permissions. Review the specific result; diagnostics alone cannot prove whether sending works. |
+| API accepted the message but nothing arrives | Inspect queued, bounced and suppressed recipient outcomes, then delivery events and the destination's spam folder. Acceptance is not final delivery. |
+| `deliver_later` does nothing | Check your job backend, running workers and failed jobs. For the outbox's `SendJob`, make sure a worker processes `mailers`. |
+| Timeout or server error | Cloudflare might have accepted the message before the error. Investigate before resending. The outbox preserves this uncertainty. |
+| Some recipients received it | Inspect each recipient; do not resend the entire partial batch. See [outbox recovery](outbox.md#reconcile-uncertainty). |
+
+To perform a deliberate real-send check:
+
+```sh
+FROM=hello@mail.example.com TO=you@example.net bin/rails cloudflare:email:send_test
+```
+
+## Incoming email
+
+| What you see | What to check next |
+| --- | --- |
+| Address receives nothing | Verify receiving-subdomain onboarding and the address route's Worker. Sending-domain verification is separate. |
+| Route provisioning refuses a subdomain | Complete Email Routing subdomain setup and DNS preflight first. Do not enable or replace apex routing just to bypass the error. |
+| Worker reports 401 from Rails | Check matching ingress secrets and that Rails and Worker both use the v2 code. Upgrade both together. |
+| Worker reports 408 | Check clock accuracy and the five-minute signing window. |
+| Worker reports 413 or rejects size | Raw MIME exceeds the configured limit. Match `MAX_EMAIL_BYTES` on Rails and Worker and check upstream request limits. |
+| Worker reports 3xx | Point it directly at the final HTTPS ingress URL; redirects are intentionally rejected. |
+| Worker times out | Check Rails availability and request latency. Its upstream request limit is 15 seconds; there is no durable inbound buffer. |
+| Rails returns 200 but your product shows no message | Check ActionMailbox records, job workers, failed routing jobs and your mailbox's `process` method. The default mailbox only logs receipt. |
+| Retrying creates no new inbound record | Identical MIME for the same SMTP recipient is intentionally deduplicated. |
+| Bcc message appears routed to the wrong mailbox | Use `Envelope.for(inbound_email)["to"]` after checking the envelope exists; MIME `To` does not identify every SMTP recipient. |
+| Quoted or internationalized addresses are rejected | The v2 envelope currently supports ASCII dot-atom addresses, not every possible email address syntax. |
+
+## Local development
+
+Restart Rails after upgrading so the ingress-only guard is installed. Start
+Rails on the expected port before `cloudflare:email:dev`, install `cloudflared`,
+and use development credentials and a development receiving route.
+
+The tunnel only permits POSTs to the email ingress. Opening its root URL in a
+browser returning 404 is expected. When you stop the tunnel, its URL remains in
+the development Worker until the next update; it no longer reaches your app.
+
+## Saved outbox operations
+
+```sh
+bin/rails cloudflare:email:pending_deliveries
+```
+
+This lists operations requiring attention for the configured account. It does
+not dispatch or repair them automatically.
+
+| State or error | What to do |
+| --- | --- |
+| `prepared` | The message is saved and can be dispatched using its existing operation key. Check for an enqueue failure or stopped worker. |
+| `sending` | A process claimed the send. Verify its status; age alone does not prove it failed to send. |
+| `unknown` | Collect provider evidence before making a resend decision. |
+| `partial` | Review individual acceptance outcomes. Already accepted recipients must not receive a batch retry. |
+| `accepted` | Repeated delivery uses the existing record without another send. Look at recipient lifecycle for later delivery results. |
+| `rejected` | Correct the cause; a deliberate new attempt needs a new operation key. |
+| `SnapshotConflict` | The same key was used with different MIME/envelope data. Retry the saved identity; do not re-render it. |
+
+For a known `prepared` operation, dispatch the saved message with:
+
+```sh
+OPERATION_KEY='your-existing-operation-key' bin/rails cloudflare:email:deliver
+```
+
+For uncertainty, follow [audited reconciliation](outbox.md#reconcile-uncertainty).
+Never reset a row to `prepared` or delete the ledger to make a retry go through.
+Reconciliation requires an authorized operator and evidence; the gem records
+the actor you supply but does not authenticate that person.
+
+## Delivery updates
+
+| What you see | What to check next |
+| --- | --- |
+| Queue stays empty | Check the Email Sending subscription, sending domain, selected event types and a real send from that domain. |
+| Polling fails before processing | Check the queue ID, separate Queues Read/Write token, HTTP pull configuration and configured event handler. |
+| Events process once then stop | `consume_events` is a one-batch task. Schedule it to run repeatedly. |
+| Events stay unmatched | Check account, normalized provider message ID and recipient. Schedule replay; a receipt can arrive before the send result is saved. |
+| Acknowledged event did not update the UI | Durable receipt storage and product projection are separate. Check replay, callback failures and application correlation. |
+| Old delivery status does not replace a newer one | Ordering and terminal-state guards intentionally reject stale updates. |
+| One event repeatedly fails | Inspect validation/handler errors and dead-letter queue policy. Invalid events are not acknowledged by the consumer. |
+
+```sh
+bin/rails cloudflare:email:consume_events
+bin/rails cloudflare:email:replay_events
+```
+
+See [delivery events](delivery-events.md) for queue setup and
+[observability](../README.md#observability-and-permissions) for notifications you
+can connect to your monitoring. A delivery notification wraps handler execution;
+it is not proof that queue acknowledgement completed.

data/docs/upgrading-0.2.md

@@ -0,0 +1,96 @@
+# Upgrade from 0.1.0 to 0.2.0
+
+0.2.0 adds delivery events, optional managed mailboxes and opt-in tenant adapters,
+and changes defaults where 0.1.0 could resend mail or misconfigure routing.
+Database multi-tenancy remains off unless explicitly configured.
+
+## Application changes
+
+1. **Ruby:** the declared minimum is now 3.2, matching the oldest compatibility job. Prefer a maintained Ruby version. CI also covers Ruby 4.0 with Rails 8.1.
+   Base64 is now an explicit runtime dependency so isolated Ruby 3.4/4 applications can load the gem without Rails' transitive dependencies.
+2. **Initializer:** replace direct credential lookups with `Cloudflare::Email::Credentials.account_id` and `.api_token`. This fixes the documented environment fallback for existing apps; upgrading the gem does not rewrite an initializer.
+3. **Send retries:** 429 and pre-send connection failures retry by default. Ambiguous failures (read/write timeouts, resets, and 5xx) no longer resend automatically. Set `retry_ambiguous: true` only if duplicate sends are an acceptable tradeoff. Review ActiveJob retries separately.
+4. **Receiving environments:** the installer now configures development and production ingress. Existing applications should add `config.action_mailbox.ingress = :cloudflare` where needed. `--all-envs` also includes test; staging is explicit.
+5. **Dev tunnel:** `cloudflare:email:dev` refuses staging/production. Use separate routed addresses and Workers for development.
+6. **Test sends:** both `FROM=` and `TO=` are required. The unsupported account-domain autodetection endpoint was removed.
+7. **JSON/Rails:** the gem's development bundle pins JSON below 3 because current tested Rails releases call the pre-3.0 positional-options API. If an existing Rails application's bundle selects JSON 3 and fails in `ActiveSupport::JSON.decode`, constrain `gem "json", "< 3"` in that application's Gemfile until its Rails version supports JSON 3. The gem does not impose this Rails-specific constraint on plain Ruby consumers.
+
+## Worker and routing migration
+
+Copy/review the updated Worker files from this gem; the generator does not silently replace your deployed Worker.
+
+0.2.0 is a preproduction protocol break: only the bundled v2 Worker protocol is
+accepted. Missing or v1 signature versions receive HTTP 401 before persistence
+or routing. Its HMAC covers the SMTP envelope as well as unchanged MIME.
+Pause test ingress while updating Rails and the deployed Worker together; verify
+the matching ingress URL and shared secret before resuming delivery. There is no
+legacy routing mode or supported mixed-version rollout.
+
+Rails stores authenticated metadata before enqueueing routing jobs.
+`Cloudflare::Email::Envelope.for(inbound_email)` returns a string-keyed `from`/`to`
+hash for accepted ingress. It returns `nil` for records from another ingress or
+older stored records without verified metadata; applications should refuse to
+route those using sender-supplied MIME headers. Route tenant mailboxes using the
+trusted SMTP recipient instead of MIME `To`/`Cc`.
+
+The [earlier upgrade rehearsal](verification/2026-09-10-install-upgrade.md) is a
+historical report and its temporary legacy-mode instructions no longer apply.
+Identical MIME for separate SMTP recipients is stored separately; retries for the
+same exact recipient remain duplicates. This includes Bcc deliveries without a
+visible recipient header. The envelope format supports ASCII dot-atom addresses
+up to 254 bytes, with a 64-byte local part, and an empty SMTP sender for bounces.
+
+- Node tooling uses a tracked package-lock, `npm ci`, Node 22.12+ (or supported newer), Wrangler 4.131+, and Vitest 5.
+- Include `scripts/wrangler.mjs`, `package-lock.json`, and the updated `wrangler.toml`.
+- Wrangler deploy/dev scripts require `--env development|staging|production`.
+- Both deployment paths now target `cloudflare-email-ingress-<environment>`. Existing unsuffixed Workers are not deleted. Set both secrets on the new Worker, deploy, verify, then deliberately update the corresponding route.
+- The forwarding Worker now times out after 15 seconds and refuses redirects. Ensure the Rails URL is the final HTTPS endpoint. Both deployment paths use compatibility date 2026-09-10.
+- For an installer `--worker-dir` other than the default, pass `SCRIPT=your-directory/src/index.js` to the Ruby deploy task.
+- The development tunnel forces a dedicated origin Host and permits only POSTs to the email ingress. Restart the Rails development server after upgrading; the tunnel task verifies the ingress guard before starting.
+
+## Security hardening
+
+- Upgrade host applications to patched Rails: tested floors are 7.2.3.2, 8.0.5.1, and 8.1.3.1, with SQLite 2.9.6 in the test stacks. Rails 7.1 is no longer supported. These are development/test constraints, not runtime dependency enforcement in your application.
+- Redeploy the bundled Worker after updating the gem. Rails and the Worker now default to a 25 MiB raw-email limit. Set `MAX_EMAIL_BYTES` to the same positive integer on both sides when overriding it. Configure upstream request limits and deadlines too.
+- API and ingress endpoints require HTTPS, with HTTP allowed only for literal loopback development hosts. Remove URL credentials, fragments, and query options from configured Ruby endpoints.
+- Delivery events require recipient, ISO8601 timestamp, a boolean `terminal`, valid identifiers, and object-shaped optional details. Invalid messages remain unacknowledged; monitor and quarantine poison messages through your queue operations.
+- Event receipt identity and original payload are read-only through normal ActiveRecord updates. Privileged database access remains trusted.
+- See [the security review](verification/2026-09-11-security.md) for evidence and limits.
+- Subdomain routes require separately onboarded routing DNS and DNS Read access for preflight. The provisioner no longer enables the parent apex to make a subdomain work.
+- Catch-all provisioning is explicitly zone-wide. A request for a subdomain that resolves to a parent zone fails rather than replacing the parent's catch-all.
+- Apex enablement uses the current routing DNS API. Permission/setup errors stop provisioning; they are no longer ignored.
+- Rule lookup now paginates and only updates matching single literal-address rules.
+
+Do not enable Cloudflare routing at the apex of a domain whose mail belongs to Workspace/Outlook. Follow Cloudflare's current subdomain onboarding flow.
+
+## Delivery state and thread correlation
+
+Responses and notifications expose `message_id` when present and `suppressed_recipients`. API success can include permanent bounces or suppression. Monitor recipient outcomes.
+
+Add a dedicated Queue, Email Sending event subscription, and HTTP pull consumer to use [delivery events](delivery-events.md). Existing applications are not subscribed or polled automatically. Provide an idempotent handler and configure retries/dead-letter handling.
+
+For durable sending, install the optional [outbox](outbox.md) migrations and use
+its saved-operation API. Ordinary `deliver_now`/`deliver_later` do not silently
+opt into durable claims. Existing receipt tables from the earlier preproduction
+adapter should normalize their stored `message_id` column before filtered replay;
+retain the raw payload. The reference inbox's migration imports its existing
+attempts, recipients and audits without clearing uncertainty. Outbox and tracking
+generators now refuse destructive rollback; plan forward fixes and backups.
+
+The old README's signed-reply identity and exactly-once claims were too strong. The unused signed-ID helper has been removed. A repeated Message-ID is not a send idempotency key. Live testing confirmed Cloudflare replaces custom IDs: store provider IDs and include parent IDs in outgoing reply headers. See [thread correlation](thread-correlation.md).
+
+## Verification before publishing/deploying
+
+Local automated coverage includes fresh Rails installation/migrations/mailbox execution, task orchestration, packaged installation, duplicate deliveries, mocked sending/routing/event APIs, Worker unit tests, dry-run builds, and actual local workerd-to-Rails forwarding. The runtime test caught and fixed unsupported `redirect: "error"`; the Worker now uses `manual` and rejects 3xx responses. The API tests exercise documented contracts; they do not prove a live account is configured. See the [verification report](verification/2026-09-10.md).
+
+In an account and mailboxes you control, verify these before production rollout:
+
+- Sending-domain DNS and outbound MIME/attachments/cc/bcc behavior.
+- Subdomain receiving routes and each environment's deployed Worker.
+- Ingress response handling and application mailbox jobs.
+- Queue subscription, event encoding, handler persistence, and acknowledgements.
+- Actual delivered provider Message-ID and reply threading.
+
+The automated suite needs no live account. Separate authorized isolated live runs
+are documented in the [live follow-up](verification/2026-09-10-followup.md).
+The gem has not been published by these verification tasks.

data/docs/verification/2026-09-10-followup.md

@@ -0,0 +1,49 @@
+# Envelope, drafting, and delivery verification — 2026-09-10
+
+The [install/upgrade rehearsal](2026-09-10-install-upgrade.md) and the
+[inbox E2E report](https://github.com/cole-robertson/agentic-inbox-rails-full/blob/verification/remaining-e2e/docs/verification/2026-09-10-e2e.md)
+extend this pass with crash/restart and live queue dead-letter/replay evidence.
+
+This follow-up supersedes the missing-envelope, missing-ledger, and maintenance-failure findings in the initial reports. The app pins gem candidate `be879439353752d2127dd6413bf915f7321e3fa6`; its runtime matches the live-tested `cb9b87a` (the later commit fixes test-fixture reloading only).
+
+## Implemented
+
+- Worker v2 authenticates SMTP sender/recipient alongside unchanged MIME. Rails stores trusted metadata before routing jobs enqueue and deduplicates separately per recipient. The app routes strictly by trusted recipient; legacy visible-To fallback requires explicit configuration.
+- Drafting supplies actual incoming content and sent history, validates structured output, rejects placeholders, retains existing drafts on failures, and exposes retry/review UI. Re-triage preserves completed tasks and avoids duplicate titles.
+- Review flags combine model output with conservative application checks for approval/financial content, commitments, clarification, and unread attachments. Auto-send and agent SendDraft refuse flagged drafts; the owner can explicitly approve in the UI. These heuristics do not guarantee semantic correctness or protect against every possible tool chain.
+- Durable send claims and immutable attempt snapshots block repeated sends after ambiguous outcomes. Operator reconciliation is scoped, documented, and audited. Event receipts deduplicate before ACK, correlate by provider ID and recipient, and handle ordering/unmatched replay.
+- Compose, UI approval, agent tools, and auto-send share delivery handling. The UI shows state, uncertainty, review reasons, and generation failures. Draft editor state resets when the selected draft changes.
+- Updated Ruby/npm dependencies and resolved the earlier lint, formatting, scanner, and audit findings.
+
+## Live evidence
+
+Only dedicated addresses under `test.rebulk.com` and an isolated SQLite database/storage were used. Real LLM inference used the configured `gpt-4o-mini`.
+
+1. A structured send to To/Cc/Bcc produced three received records sharing the same visible To and Message-ID. Each authenticated SMTP recipient routed to its own mailbox and separate conversation, including the hidden Bcc mailbox.
+2. Four received messages were processed with real triage/drafting. Replies referred to the blue sample box; none contained the earlier scaffolding or placeholders.
+3. A purchase/next-day commitment scenario initially returned a sensible clarification but an incorrect model review flag. Added independent application rules and a regression. The rerun required review, included concrete reasons, and stayed unsent.
+4. An explicitly approved synthetic draft went through the real ActionMailer adapter. Its durable attempt became accepted and then delivered after its provider event was consumed. Cloudflare replaced the custom signed Message-ID; parent reply headers were preserved.
+5. The dedicated queue produced four delivered events (three direct-client recipients plus the app send). A deliberate exception after durable storage caused redelivery; the retry retained four unique receipts and ACKed all four. The app event matched its attempt/recipient; direct-client events remained stored unmatched.
+
+The first simultaneous ingress requests reached a cold, reload-enabled temporary test harness: two returned routing errors while one succeeded. After routes loaded, a new send produced all three successful copies. This was a harness startup limitation, not counted as a passing delivery. Use eager loading/disabled reloading and warm the receiver before concurrent live tests.
+
+The earlier live report separately covers raw MIME, binary attachment integrity, provider-ID reply round trips, and six initial delivered events.
+
+## Automated verification and limits
+
+Gem: 197 tests / 631 assertions; 19 Worker tests; 19 real local workerd-to-Rails checks. All 13 CI jobs passed at be87943, including 11 Ruby/Rails combinations:
+https://github.com/cole-robertson/cloudflare-email/actions/runs/34531698927
+
+The app suite passes **213 examples**, seed 42422. It covers authenticated ingress, strict mailbox selection, approval endpoints, review/auto-send guards, rejection and ambiguity, process/persistence failure, attempt snapshots, operator authorization, ordering, deduplication, event task wiring, and malformed LLM responses. Lint, TypeScript, Prettier, RuboCop, Brakeman and Ruby/npm audits pass (zero reported vulnerabilities/warnings).
+
+Browser acceptance was attempted using the shared T3 preview: status/open repeatedly timed out, including a fresh-tab attempt. No visual browser pass is claimed; authenticated request tests and frontend static checks passed.
+
+Only delivered lifecycle events were induced live. Bounce, suppression, complaint, and deferred/failure handling have automated coverage. Queue dead-letter exhaustion was not exercised; no exhaustive provider failure simulation or spam complaints were generated. The token lacks DNS Read, so complete route-provisioning preflight still needs an appropriately scoped credential. Exact test rules were created through Email Routing after the prior DNS verification. No exactly-once guarantee is possible across provider/database boundaries; separate compose submissions are distinct messages.
+
+## Cleanup and reproduction
+
+Deleted the four exact test routes, dedicated Worker, event subscription, and queue. Existing routing rules and catch-all exactly match the saved baseline. Stopped the local receiver/tunnel; no DNS changes, production migration, merge, or gem publication occurred.
+
+Private harness/evidence: `/tmp/cf-email-envelope-live-20260911` (the suffix is a unique run identifier; execution was September 10). No credentials or queue leases are in tracked evidence. Sanitized observations are in [2026-09-10-followup-results.json](2026-09-10-followup-results.json).
+
+Run `bundle exec rspec`, `bundle exec rubocop`, `bin/brakeman --no-pager`, `bin/bundler-audit`, `npm run lint`, `npm run check`, `npm run format`, and the checked-in Worker tests/build. See the app's delivery operations guide for event consumption and operator reconciliation. Deploy the new Rails gem and app migrations before the v2 Worker.

data/docs/verification/2026-09-10-install-upgrade.md

@@ -0,0 +1,86 @@
+# Install, upgrade, rollback, and staged protocol rehearsal
+
+Executed 2026-09-10 with Ruby 4.0.0, Rails 8.1.3.1, gem 0.2.0.
+Gem merge: `cb8d8932296f64b5531ee2e1b773e9937c6df884`.
+Inbox merge: `21bc3627ad70c586c45e7df297db4af4cd070066`.
+The inbox retains its release-candidate gem pin `be879439353752d2127dd6413bf915f7321e3fa6`.
+
+## Packaged install
+
+```sh
+bundle exec ruby script/verify_package.rb
+```
+
+Passed: isolated consumer installed the built `.gem` into a temporary bundle,
+loaded version 0.2.0, and did not load Rails. Rails integration fixtures loaded
+the extracted packaged files in separate processes:
+
+| Fixture | Tests | Assertions | Failures/errors/skips |
+| --- | ---: | ---: | --- |
+| Inbound | 18 | 95 | 0/0/0 |
+| Send only | 11 | 40 | 0/0/0 |
+| Fresh inbound | 19 | 100 | 0/0/0 |
+
+The fresh Rails harness invoked the actual install generator, installed and
+ran Active Storage/Action Mailbox migrations, generated mailbox files and Worker
+files, then received and routed a signed message. This verifies the packaged
+generator in a minimal fresh Rails application; it does not claim a RubyGems
+publication, Docker deployment, or a production installation.
+
+## Existing inbox migration and backup recovery
+
+From the gem checkout, using the inbox bundle:
+
+```sh
+INBOX_ROOT=/absolute/path/to/agentic-inbox-rails-full \
+BUNDLE_GEMFILE=/absolute/path/to/agentic-inbox-rails-full/Gemfile \
+bundle exec ruby script/verification/install_upgrade.rb
+```
+
+All **16 checks passed**. The script creates temporary SQLite databases and
+storage, uses inert test credentials, disables mail delivery, and blocks network
+access during routing. It removes those resources on exit and never migrates the
+configured inbox database.
+
+- Migrate a completely empty database to the pre-ledger migration
+  `20260418012251`; insert representative incoming, sent, and unapproved messages.
+- Upgrade to the current schema; preserve every original message field and
+  timestamp. Previously sent messages become `legacy_sent`, while unsent messages
+  become `pending`. No provider IDs or delivery attempts are fabricated.
+- Check review-field defaults on existing rows.
+- Insert an ambiguous send and durable attempt. Take a quiesced SQLite snapshot
+  with `VACUUM INTO`, roll back both new migrations, and reapply them.
+- Restore the snapshot into a different database; verify the original ambiguous
+  attempt and message state survive. Re-running migrations is idempotent, and an
+  actual `DraftDelivery.call` remains blocked without creating another attempt.
+
+**Schema rollback is destructive to delivery history.** The rehearsal confirmed
+that down/up migrations preserve old message content but drop attempts and reset
+an ambiguous draft to `pending`. A schema rollback after send attempts exist must
+not be treated as a safe retry/recovery procedure. Stop writers and retain the
+ledger; prefer a forward fix. If restoring a backup, coordinate ingress, jobs,
+and provider reconciliation first: the local quiesced backup test does not prove
+that restoring an earlier production snapshot is safe while external sends are
+in flight. Rolling back to old sending code can also bypass the new ledger guard.
+
+## Rails-before-Worker rollout
+
+The same script POSTs correctly signed v1/v2 payloads to the real Rails ingress
+and routes them through `AgentMailbox`. A closed, existing conversation avoids
+LLM requests and outbound mail.
+
+| Protocol and configuration | Observed result |
+| --- | --- |
+| v1, `ALLOW_LEGACY_EMAIL_ROUTING=true`, single To/no Cc | Accepted and routed into the existing conversation |
+| v1, transitional mode, Cc present | Ingress accepts; mailbox refuses ambiguous routing and records bounced |
+| v1, strict mode | Ingress accepts; mailbox refuses missing authenticated envelope and records bounced |
+| v2, strict mode | Accepted and routed into the existing conversation |
+
+Only the two permitted messages enter the conversation; no outbound mail is sent.
+For a staged rollout, deploy Rails with the explicitly temporary legacy flag,
+deploy the v2 Worker, verify authenticated-envelope delivery, then disable the
+flag. Legacy multi-recipient traffic is intentionally refused even during this
+transition; use a controlled ingress pause if this traffic must not be interrupted.
+The protocol replay tests exercise real HMAC verification, persistence, and
+mailbox execution, with synthetic local HTTP requests; they are not additional
+Cloudflare-deployed Worker tests.

data/docs/verification/2026-09-10-live.md

@@ -0,0 +1,102 @@
+# Live verification — 2026-09-10
+
+> Initial live pass. The [follow-up](2026-09-10-followup.md) resolves the
+> envelope-routing, missing-ledger, and maintenance findings below, improves
+> drafting/review behavior, and records new live evidence and remaining limits.
+
+The updated gem was exercised against the real Cloudflare account owning
+`rebulk.com`, using only isolated addresses under its existing enabled sending
+subdomain `test.rebulk.com`. The Rails consumer was `agentic-inbox-rails-full`,
+with a private temporary SQLite database/storage and a restricted loopback
+server exposed through a temporary tunnel. Existing production resources and
+the old `cole@test.rebulk.com` experiment were preserved.
+
+## Results
+
+| Check | Live evidence |
+|---|---|
+| Structured send | One message addressed To/Cc/Bcc: three queued recipients, three actual received copies, three delivered events |
+| MIME headers | Text and HTML received, Cc retained, Bcc absent from MIME headers; sending-domain DKIM, SPF and DMARC passed |
+| Raw MIME | One real send with a 260-byte binary attachment; received and application-stored SHA-256 matches `0d42a8b2489dc6e5632cec0b1f06b4f41b61742536d2c3c361edb990e3b945cd` |
+| Deployed Worker | Gem `WorkerDeployer` uploaded the shipped Worker and set both secrets; all six received copies reached real Rails HMAC ingress |
+| Reply round trip | Two explicitly approved synthetic draft sends through real `DraftDelivery`/ActionMailer; returned reply matched the original conversation using the stored provider ID |
+| Custom signed Message-ID | **Not preserved.** Raw send and ActionMailer sends were assigned Cloudflare IDs; received IDs matched provider responses |
+| Actual LLM | Four inbound messages ran real triage and drafting; categories, summaries, action items and drafts were persisted; auto-send stayed off |
+| Ingress retries | Exact signed replay returned 200 without a new record; invalid signature returned 401 without persistence |
+| Delivery events | Six unique `delivered` events persisted and acknowledged through the real HTTP pull API; subsequent poll returned zero |
+| Handler failure/redelivery | Deliberate exception after durable event insert prevented acknowledgement; next poll redelivered the same event, inserted zero duplicate rows, then acknowledged it |
+| Isolation/cleanup | Four temporary literal rules, one Worker, one event subscription, two queues and local server/tunnel removed; original eight rules and catch-all exactly matched the saved baseline |
+
+Four outbound API calls produced six recipient deliveries. No mail was sent to
+people or production ingestion addresses. No DNS records were changed, and the
+apex Google Workspace MX configuration was not touched.
+
+## Defects found and fixed
+
+1. **Queue JSON encoding.** Real Email Sending subscriptions return plain JSON
+   strings with `CF-Content-Type: json`. The consumer previously assumed Base64,
+   so the first live poll failed without acknowledging the message. Commit
+   `3fda75aa7cd5c54fc7753676e340d1deef4f0ce4` accepts plain JSON and keeps Base64
+   compatibility. A sanitized live-shaped fixture and regression checks were
+   added. The fixed consumer passed real redelivery/deduplication/acknowledgement.
+2. **App reply headers.** The demo stored provider IDs but its outgoing replies
+   omitted `In-Reply-To` and `References`. Its follow-up change sets both from
+   the latest inbound message, and the real round trip then correlated correctly.
+
+Follow-up automated results: **193 gem tests / 602 assertions pass**, plus all
+13 jobs in [gem CI](https://github.com/cole-robertson/cloudflare-email/actions/runs/34527748184).
+The updated inbox suite passes **144 examples** with eager loading (seed 38933).
+The inbox now pins the fixed gem commit above in
+[PR #26](https://github.com/cole-robertson/agentic-inbox-rails-full/pull/26).
+
+## Boundaries and remaining findings
+
+- **Inbound Cc/Bcc mailbox selection is not implemented by the demo.** All three
+  actual copies retain the same visible To header, while their SMTP recipient
+  differs. The Worker forwards raw MIME without a separately authenticated
+  envelope recipient; `AgentMailbox` selects the first visible To address.
+  Only the primary To copy was processed in this test. Correct multi-mailbox
+  Cc/Bcc routing requires an explicit envelope-aware design; do not infer it
+  from successful outbound Cc/Bcc delivery.
+- **LLM response quality is not guaranteed.** Real replies sometimes included
+  drafting instructions/placeholders; one asked for context that was already
+  available. This pass proves live inference and persistence, not acceptable
+  autonomous reply quality. Human draft review remains necessary.
+- Wrangler OAuth lacks DNS Read. The gem's subdomain provisioning preflight
+  correctly refused that credential before creating rules. Public DNS confirmed
+  the three Cloudflare MX records; the four isolated literal rules were then
+  created directly using the authorized Email Routing API. The complete
+  `provision_route` path still needs a suitably scoped token for live validation.
+- Only `delivered` lifecycle events were induced live. Bounce, suppression,
+  complaint, and deferred/failure cases remain automated
+  fixture coverage. Dead-letter exhaustion was not tested. No spam complaints or deliberately harmful traffic were
+  generated. A dead-letter queue was configured but its exhaustion path was not
+  exercised.
+- The gem's development-tunnel orchestration task was not used; this run used an
+  explicit temporary tunnel with a restricted receiver. Arbitrary external mail
+  providers/clients, SMTP submission, exhaustive browser UI behavior and every
+  agent/MCP tool remain outside this live pass.
+- The app's existing dependency-audit/style failures and missing durable
+  ambiguous-send ledger remain as documented in its earlier report. No release
+  publication, production migration or merge occurred.
+
+## Reproduction and evidence
+
+Use an isolated app database and dedicated exact receiving addresses. Snapshot
+existing rules/catch-all; deploy a uniquely named shipped Worker; set its HMAC
+secret/receiver URL; provision only the dedicated addresses. Create an HTTP-pull
+queue and a domain-scoped `email.sending` subscription before sending.
+
+Exercise `Client#send`, `Client#send_raw`, the app's `DraftDelivery`, and
+`EventConsumer#poll`. Persist event IDs under a unique index before acknowledging;
+deliberately raise once after commit to test idempotent redelivery. Compare actual
+received headers/attachment hashes and conversation IDs. Wait for delivered
+events, then remove only the resources created by the run and compare the saved
+baseline. Do not rerun live sends without confirming the active target scope.
+
+Sanitized machine-readable observations are in
+[2026-09-10-live-results.json](2026-09-10-live-results.json). Detailed runner
+scripts, synthetic MIME, local database and private logs remain in
+`/tmp/cf-email-live-20260910` on the verification host; they are not committed.
+No tokens, OAuth callback codes, HMAC secrets, queue leases or LLM keys are in
+the tracked artifacts.

data/docs/verification/2026-09-10.md

@@ -0,0 +1,175 @@
+# Verification report — 2026-09-10
+
+## Result
+
+The local automated checks pass after fixing a defect found only in the actual Workers runtime. This report records the initial local-only phase. A subsequent [live verification pass](2026-09-10-live.md) verified isolated sending, deployed ingress, replies, real LLM processing, and delivery-event redelivery/acknowledgement; it found and fixed a queue decoder bug and confirmed custom Message-IDs are replaced. See that report for current counts and remaining limits.
+
+**Rebulk does not currently dogfood this gem.** Its existing email tests pass, but a test-only switch to the gem exposes four migration blockers. Do not treat those characterization results as production acceptance.
+
+**Additional dogfooding app located and tested:** `cole-robertson/agentic-inbox-rails-full`
+is the Rails/Inertia inbox prototype built in April. [PR #26](https://github.com/cole-robertson/agentic-inbox-rails-full/pull/26)
+initially pinned this gem's 0.2.0 candidate at `a1f80d8` and passed 140 local Rails examples,
+including signed ingress → triage/draft → UI send approval → provider acceptance →
+reply threading. Its copied Worker passes nine tests and a dry-run build. See its
+[application verification report](https://github.com/cole-robertson/agentic-inbox-rails-full/blob/maintenance/cloudflare-email-0.2-dogfood/docs/verification/2026-09-10.md)
+for fixes, existing app maintenance failures, and remaining live-verification limits.
+The smaller `agentic-inbox-rails` demo remains on 0.1.0 and was not changed.
+
+Scope and source snapshots:
+
+- Gem implementation baseline: [9d1cd58](https://github.com/cole-robertson/cloudflare-email/commit/9d1cd581489690c5befbfaadc7f88f3535a3435f), plus this report's accompanying runtime fix and verification scripts.
+- Rebulk: [46db134](https://github.com/Rebulk/rebulk-system/commit/46db134e2722568d09fc1c4139266a937a03eaef), checked out separately for testing. No tracked Rebulk files changed.
+- Ruby 3.4.1 / Rails 8.1.3.1 for the final gem run; Ruby 3.4.5 / Rails 8.1.3.1 for Rebulk. Local runtime: Node 22.23.1, Wrangler 4.131.0 and its bundled workerd.
+
+## Evidence and results
+
+| Layer | Result | What it establishes |
+|---|---|---|
+| Gem suite | **192 tests, 577 assertions, zero failures/errors/skips** | Sending/MIME, retry/error contracts, response outcomes, signing, routing, tasks, events, and three subprocess Rails fixtures |
+| Rails fixture detail | **40 nested tests** across inbound, send-only, and fresh-inbound apps | Actual boot, generators, migrations, credentials, persistence, duplicate acknowledgement, default mailbox routing/job execution, and task/dev behavior; these run inside the three suite wrapper tests |
+| Actual local workerd → Rails | **16 checks passed** | Runtime interoperability, raw MIME/binary integrity, duplicate acceptance, mailbox processing, misconfiguration rejection, redirects, and request timeout |
+| Worker Node tests | **9 tests passed** | Unit behavior, including explicit 302 response handling; complementary to workerd |
+| Packaged distribution | **Isolated Ruby consumer passed; 40 Rails tests / 179 assertions passed** | Build/install of the actual .gem, Base64 dependency without Rails, and all three Rails fixture modes against packaged files |
+| Expanded CI | **[13/13 jobs passed on 4b968f6](https://github.com/cole-robertson/cloudflare-email/actions/runs/34515534609)** | Eleven Ruby/Rails combinations, packaged installation, Worker unit/build/audit, and actual local workerd → Rails |
+| Rebulk existing email suite | **437 examples in 22 files; zero failures/pending/errors outside examples** | Current SMTP-oriented mailers, custom inbound controller/COA flow, email records, retention, retries, and tracking |
+| Rebulk adapter characterization | **6 examples reproduced expected observations** | Actual UserMailer and CSV ReportMailer can use the adapter in a test process; four production migration gaps below remain |
+| Public DNS read | Completed | Apex MX is Google; mail/in/agent/test subdomains resolve to Cloudflare MX. This does not establish Worker route or token configuration |
+
+The gem suite's 192 top-level tests include three process-wrapper tests; the 40 nested Rails tests are reported separately to avoid implying they are counted by Minitest's top-level summary.
+
+### Runtime defect found and fixed
+
+The April-to-0.2 update used `fetch(..., { redirect: "error" })`. Node accepted it, so the unit tests and dry-run bundle were green. Actual workerd rejected that option before contacting Rails.
+
+The Worker now uses `redirect: "manual"` and rejects the returned 3xx response through its non-2xx path. The new runtime test proves a 302 target is never contacted.
+
+The same test:
+
+- Sends a synthetic multipart message through the real local email simulator.
+- Compares the stored raw RFC822 bytes exactly.
+- Verifies an attachment containing all 256 byte values plus binary terminators.
+- Checks a repeated message stores only one record.
+- Executes the real ActionMailbox routing job and checks the mailbox receives the attachment.
+- Checks wrong/missing shared-secret configurations do not store mail.
+- Measures the actual abort of a stalled local ingress: **15.08 seconds** in the final run.
+
+This test uses loopback HTTP and temporary storage. It does not exercise Cloudflare's managed SMTP service, deployed Workers, DNS routing, or sender-provider behavior.
+
+## Was there an old test report?
+
+I found meaningful historical tests, but no preserved, sanitized live-results report:
+
+- Local `2026-04-16-cloudflare-mailer/trial/test/integration/` contains **20 Rails integration tests** across credentials, inbound, instrumentation, outbound, and routing. Outbound uses WebMock; inbound uses local signed requests. These are not live Cloudflare tests.
+- `trial2/app/mailboxes/main_mailbox.rb` contains the signed-Message-ID reply experiment. No reproducible live round-trip assertion script was found alongside it.
+- Both `trial/` and `trial2/` are gitignored and untracked; they were never part of repository CI.
+- The original changelog/README claim live end-to-end testing and a 191-character signed Message-ID. Those are historical assertions, not current verification evidence.
+- A deleted historical `PLAN.md` was a design/checklist, not a results report.
+
+This report and the scripts below are tracked so the current evidence can be rerun.
+
+## Rebulk's actual integration
+
+At the pinned Rebulk commit:
+
+| Area | Current implementation |
+|---|---|
+| Dependency | No `cloudflare-email` entry in Gemfile or Gemfile.lock |
+| Outbound | `:smtp` in [production.rb](https://github.com/Rebulk/rebulk-system/blob/46db134e2722568d09fc1c4139266a937a03eaef/config/environments/production.rb#L72), defaulting to AWS SES; deployment environment can override SMTP settings |
+| Inbound | Custom [email-worker.js](https://github.com/Rebulk/rebulk-system/blob/46db134e2722568d09fc1c4139266a937a03eaef/cloudflare/email-worker.js) → Basic-authenticated `/api/v1/ingest/email` |
+| Routing/business behavior | Org/site routing, sender-authentication metadata, BOL/COA processing, raw R2 archive, ops recovery and backend dispatch |
+| Gem experiments | [cloudflare/README.md](https://github.com/Rebulk/rebulk-system/blob/46db134e2722568d09fc1c4139266a937a03eaef/cloudflare/README.md) distinguishes literal routes under in/agent/test.rebulk.com as separate gem experiments |
+
+The gem's HMAC ActionMailbox ingress is not a drop-in replacement for Rebulk's custom endpoint. Its existing R2 archive, tenant/site dispatch, authenticated-sender policy, recovery behavior, and BOL/COA processing would need an explicit integration design.
+
+No tests referencing Rebulk's Worker source were found in its tracked tree. The 437-example baseline exercises its Rails side; it does not establish that its deployed Worker matches the checked-in source.
+
+### Rebulk migration blockers demonstrated locally
+
+The [portable adapter probe](../../script/rebulk_outbound_probe_spec.rb) loads the gem only into a test process, uses real Rebulk mailers/observers and isolated test databases, and mocks every Cloudflare request.
+
+1. **Provider identity/correlation is wrong.** The preflight CommunicationMessage retains its original local Message-ID after the adapter receives a provider ID. Its provider remains SES/SMTP. The observer finds the record through an internal header, but does not update provider identity. See [CommunicationsEmailTracker](https://github.com/Rebulk/rebulk-system/blob/46db134e2722568d09fc1c4139266a937a03eaef/app/mailers/communications_email_tracker.rb) and [Recorder](https://github.com/Rebulk/rebulk-system/blob/46db134e2722568d09fc1c4139266a937a03eaef/app/services/communications/recorder.rb).
+2. **All-bounced/all-suppressed reports become “sent.”** A normal HTTP-success response is sufficient for Rebulk's observers to mark the report sent and its communication accepted, even when no recipient was delivered or queued. Integrate the per-recipient response/event outcomes before using this provider. See [ReportEmailDeliveryObserver](https://github.com/Rebulk/rebulk-system/blob/46db134e2722568d09fc1c4139266a937a03eaef/app/mailers/report_email_delivery_observer.rb).
+3. **Ambiguous failures lose their special treatment.** Rebulk's [ReportMailer](https://github.com/Rebulk/rebulk-system/blob/46db134e2722568d09fc1c4139266a937a03eaef/app/mailers/report_mailer.rb#L1805) recognizes raw network exception classes. The gem wraps these in `Cloudflare::Email::NetworkError`, so a read timeout moves its report claim back to queued rather than preserving the existing unknown-outcome state.
+4. **Job retries are SMTP-specific.** An exhausted `Cloudflare::Email::RateLimitError` is re-raised without scheduling an application retry. The [retry initializer](https://github.com/Rebulk/rebulk-system/blob/46db134e2722568d09fc1c4139266a937a03eaef/config/initializers/mailer_retry.rb) needs a deliberate Cloudflare policy that distinguishes pre-send/retryable errors from ambiguous delivery.
+
+The probe's green result means it reproduced these observations, **not** that a migration passes acceptance tests. Production dependencies/configuration were not changed.
+
+## Reproduction
+
+### Gem suite and package
+
+From this gem checkout:
+
+```sh
+bundle install
+bundle exec rake test
+bundle exec ruby script/verify_package.rb
+```
+
+The package script builds a temporary .gem, installs it and Base64 into an isolated consumer bundle from local gem caches, then runs the Rails fixtures against extracted packaged files. It requires a preceding `bundle install` and does not publish anything.
+
+### Actual Workers runtime
+
+```sh
+# Node 22+ required
+npm ci --prefix templates/worker
+npm test --prefix templates/worker
+BUNDLE_GEMFILE=gemfiles/local_ingress.gemfile bundle install
+BUNDLE_GEMFILE=gemfiles/local_ingress.gemfile bundle exec ruby script/verify_local_ingress.rb
+```
+
+Set `NODE_BINARY=/absolute/path/to/node` if your PATH points to an older Node. The final local run used the Node 22.23.1 executable explicitly because this host's npm wrapper otherwise selected Node 20.
+
+### Rebulk baseline
+
+Use a separate checkout of the pinned Rebulk commit, never the shared primary clone. With dependencies installed:
+
+```sh
+env -u DATABASE_URL -u RAILS_MASTER_KEY RAILS_ENV=test \
+  NIGHTRAIL_TOKEN= LANTERN_TOKEN= SECRET_KEY_BASE_DUMMY=1 \
+  bundle exec rails db:test:prepare
+
+env -u DATABASE_URL -u RAILS_MASTER_KEY RAILS_ENV=test \
+  NIGHTRAIL_TOKEN= LANTERN_TOKEN= SECRET_KEY_BASE_DUMMY=1 \
+  bundle exec rspec spec/mailers \
+    spec/jobs/mail_delivery_retry_spec.rb \
+    spec/jobs/prune_inbound_emails_job_spec.rb \
+    spec/tasks/retry_failed_mailer_jobs_spec.rb \
+    spec/requests/api/v1/email_ingest_spec.rb \
+    spec/requests/api/v1/email_ingest_coa_flow_spec.rb \
+    spec/models/inbound_email_spec.rb \
+    spec/models/report_email_spec.rb \
+    spec/models/organization_email_domain_spec.rb \
+    spec/services/communications/recorder_spec.rb
+```
+
+Observed: seed **46556**, 437 examples, 0 failures, 0 pending; 51.88 seconds of examples plus 9.91 seconds loading.
+
+The separate adapter probe's header documents its commands. Reserve `TEST_ENV_NUMBER=9` exclusively for that probe; never run another process against the same primary/tenant test databases. The adapted probe passed its six characterization examples with seed **45808**.
+
+### Public DNS observation
+
+A read-only MX lookup returned:
+
+- `rebulk.com`: Google's five Workspace MX hosts.
+- `mail.rebulk.com`, `in.rebulk.com`, `agent.rebulk.com`, `test.rebulk.com`: `route1/2/3.mx.cloudflare.net` with priorities 30/61/91.
+
+No DNS records were changed. These answers do not identify which Worker a literal address reaches or prove sending-domain verification.
+
+## Initial live acceptance gate (see subsequent live report)
+
+Do not describe the release as live-verified until these have evidence from a controlled account and confirmed recipient inbox:
+
+| Check | Required evidence |
+|---|---|
+| Account, sending domain, tokens | Read-only confirmation of the exact account/domain and least-privilege token capabilities |
+| Outbound structured + raw MIME | Provider response and received synthetic message; HTML/text, attachment hash, cc/bcc behavior, provider ID |
+| Inbound deployed Worker | Dedicated exact test address reaches an isolated verification Worker and Rails app; one stored/processed message |
+| Reply threading | Delivered raw Message-ID and returned reply headers; confirm whether custom signed IDs survive or use provider-ID correlation |
+| Delivery events | Actual domain-scoped Queue subscription, observed event, durable handler result, successful acknowledgement, duplicate-safe processing |
+| Environment isolation | Confirm exact Worker names/addresses and secrets; no production route changes |
+| Cleanup | Remove only newly created verification resources; retain sanitized results |
+
+Proposed live scope for approval: a dedicated address under the already-existing `test.rebulk.com`, a uniquely named verification Worker/Queue, a temporary test Rails app, and a mailbox explicitly confirmed by the owner. Inspect existing address routes first; do not replace Rebulk's zone catch-all or existing experimental routes.
+
+These steps send real email and may create/deploy isolated resources, so the target mailbox/account and live scope must be confirmed before execution. No real email, Cloudflare deployment, Queue consumption, secret decryption, or production changes occurred during this verification. Public DNS lookup was the only live infrastructure inspection.