cloudflare-email 0.1.0 → 0.3.0

data/docs/mailboxes.md

@@ -0,0 +1,363 @@
+# Create and manage mailboxes from Rails
+
+The optional mailbox module lets your app create mailboxes and aliases, receive
+mail into them, track read/archive state, and send through the durable outbox.
+It works in one SQLite database or with a separate database per organization.
+It supplies models and services; your app supplies permissions and the UI.
+
+This module is available since version 0.2.0. Install `gem "cloudflare-email", "~> 0.3.0"` for the latest features.
+
+**Database multi-tenancy is off by default.** The mailbox generator works with
+one ordinary database. Calling `for_tenant` groups and scopes mailbox records;
+it does not create databases or install a tenant adapter. Separate databases
+require explicit `Tenancy.configure(...)` before models load. Neither Rails nor
+`activerecord-tenanted` is added to plain Ruby applications by this module.
+
+## Install
+
+First configure [sending and receiving](getting-started.md). Then run:
+
+```sh
+bin/rails generate cloudflare:email:mailboxes
+bin/rails db:migrate
+```
+
+The generator includes the outbox and tracking generators. Do not generate those
+migrations a second time if you already installed them; inspect existing
+generator conflicts and retain your installed migrations. Restart Rails after
+configuration changes. The generated initializer explicitly loads:
+
+```ruby
+require "cloudflare/email/mailboxes"
+```
+
+Loading this module enables mailbox lookup on the gem's Cloudflare ingress.
+Existing receiving addresses must be registered and activated before switching
+an existing application over. Unregistered or suspended destinations return
+HTTP 422 and the Worker rejects the delivery by default. You can explicitly
+configure a domain catch-all below when your application needs that behavior.
+
+For separate organization databases, follow the
+[activerecord-tenanted setup](activerecord-tenanted.md) **before loading the
+models**. It covers shared versus tenant migrations, tenant-specific framework
+storage, early initialization and background jobs. Rails and ActiveStorage must
+use the same tenant connection as the mailbox tables for atomic incoming storage.
+
+## Register an organization's domain
+
+Run directory management from your authorized administration/provisioning code.
+Do not expose arbitrary domain claims to customers without ownership checks.
+
+```ruby
+inboxes = Cloudflare::Email::Mailboxes
+domain = inboxes.register_domain(
+  domain: "acme.example.com",
+  tenant_key: "organization-123",
+  account_id: Cloudflare::Email::Credentials.account_id,
+)
+```
+
+This creates a **pending directory entry**, not DNS records. Set up the receiving
+subdomain in Cloudflare and deploy the Worker. Then record your verification:
+
+```ruby
+inboxes.activate_domain!(domain.id,
+  evidence: "Receiving DNS and the development Worker verified in setup ticket 42",
+  sending_enabled: false,
+)
+```
+
+Activation records an operator assertion; the gem does not independently verify
+domain ownership from this string. Set `sending_enabled: true` only after
+Cloudflare has also verified this sending domain. Incoming routing and outgoing
+domain verification are separate setup steps. An address's route is activated
+separately below.
+
+Each exact domain belongs to one tenant. Domain ownership, tenant key and
+Cloudflare account identity cannot be reassigned through normal model updates.
+Keep tenant keys stable and resolve them to provisioned databases through your
+trusted tenant adapter. Use customer/organization IDs rather than deriving a
+database filename directly from an email address.
+
+## Create a mailbox and activate its route
+
+Even in a single database, use an explicit tenant key and the scoped session:
+
+```ruby
+inboxes.for_tenant("organization-123") do |account|
+  mailbox = account.create(
+    name: "Customer support",
+    address: "support@acme.example.com",
+    owner_ref: "team:42",
+  )
+  address = account.addresses(mailbox.id).first
+
+  provisioner = Cloudflare::Email::RoutingProvisioner.new(
+    api_token: Cloudflare::Email::Credentials.management_token,
+  )
+  account.provision_address!(address.id,
+    provisioner: provisioner,
+    worker_name: "cloudflare-email-ingress-development",
+  )
+end
+```
+
+`owner_ref` is your application's optional stable reference to a user, site,
+team or customer. The gem stores it; it does not load that record or authorize
+the person using it. Authorize the organization and mailbox in your app first.
+
+The address stays pending if provisioning fails. The provisioner upserts an
+individual address rule, so you can retry setup. Run provisioning outside a
+database transaction. DNS propagation and an actual inbound test are still
+needed after the API succeeds.
+
+If you already configured a route separately, record that explicitly:
+
+```ruby
+account.activate_address!(address.id, evidence: "Existing route verified in ticket 43")
+```
+
+This example assumes the same `for_tenant` block and local variables as above.
+The gem does not change a zone-wide catch-all while creating a mailbox.
+
+## Aliases, suspension and ownership
+
+An alias is another address attached to the same mailbox:
+
+```ruby
+inboxes.for_tenant("organization-123") do |account|
+  mailbox = account.mailboxes.find_by!(owner_ref: "team:42")
+  alias_address = account.add_address(mailbox.id, address: "help@acme.example.com")
+  # Provision alias_address exactly as you provisioned the first address.
+
+  account.suspend(mailbox.id) # Stops new ingress and queued mailbox sends.
+  account.resume(mailbox.id)
+  # To suspend only one address:
+  account.suspend_address(mailbox.id, alias_address.id)
+end
+```
+
+Suspension is enforced by Rails; it does not delete the Cloudflare route. Mail
+already accepted and stored remains available for processing and inspection.
+Use `activate_address!` with new evidence to reactivate an address. Domain-wide
+suspension is an administrative update of `ReceivingDomain#state` to `suspended`.
+
+Addresses use lowercase ASCII dot-atom local parts and domains. Alias addresses
+are explicit: the module does not automatically strip `+tags` or invent address
+fallbacks. Reserve application-specific names such as Rebulk's `tracking` in
+your mailbox-management policy before creation.
+
+## Receive unregistered local parts with an optional catch-all
+
+Catch-all receiving is **off by default** and is available in the 0.3
+version. It lets one existing address receive otherwise unregistered addresses
+on its exact domain, without creating an alias row for each incoming local part.
+For example, `anything@acme.example.com` can arrive in an existing support mailbox.
+It does not cover subdomains such as `anything@other.acme.example.com`.
+
+Fresh installations include the schema. For an existing installation, generate
+the optional upgrade and apply it to the database containing your mailbox tables:
+
+```sh
+bin/rails generate cloudflare:email:mailboxes:catch_all
+bin/rails db:migrate
+```
+
+With separate tenant databases, pass
+`--tenant-migrations-path=db/tenant_migrate` and run your application's tenant
+migration command for every affected tenant. Updating the gem alone keeps older
+schemas working with exact-address routing. Enabling a catch-all without its
+migration raises a configuration error.
+
+First configure and verify the domain's catch-all Email Routing rule in Cloudflare
+to send mail to your Worker. Then use your authorized provisioning code:
+
+```ruby
+Cloudflare::Email::Mailboxes.for_tenant("organization-123") do |account|
+  mailbox = account.mailboxes.find_by!(owner_ref: "team:42")
+  address = account.addresses(mailbox.id).find_by!(address: "support@acme.example.com")
+  account.enable_catch_all(address.id,
+    evidence: "Domain catch-all Worker route verified in setup ticket 57")
+
+  # Later, stop accepting unregistered local parts:
+  # account.disable_catch_all(address.id)
+end
+```
+
+The address, mailbox and receiving domain must be active before enabling it.
+The evidence is an operator assertion, as with ordinary address activation;
+this call does not change Cloudflare rules. Only one active address per domain
+can serve as a catch-all. A database constraint also protects concurrent changes.
+Suspending that address pauses fallback. If another catch-all is enabled while
+it is suspended, disable the old flag before reactivating the old address.
+
+Exact registered addresses always take precedence. An exact address that is
+pending, suspended, or belongs to a suspended mailbox is rejected; it does not
+fall through to another mailbox. The catch-all also stops receiving when its
+mailbox or domain is suspended.
+
+Both `with_recipient` and `receive` yield `destination.catch_all == true` only
+when fallback was used. `destination.recipient` remains the actual envelope
+recipient, while `address_id` identifies the existing backing address. Stored
+mailbox membership retains that actual recipient too. Apps can use this flag to
+enforce additional rules, such as allowing fallback only while an organization
+has one site, before accepting/persisting the message.
+
+Catch-all receiving grants **no additional sending addresses**. Outbound From,
+Sender and envelope sender must still match an exact active registered address.
+Disable catch-all receiving through the API; the management UI displays its
+configuration but does not enable or disable it.
+
+## Read incoming mail
+
+The Worker signs the SMTP recipient. Rails verifies the whole request, resolves
+the domain, enters its tenant context and stores raw mail plus a mailbox
+membership in one database transaction. MIME To/Cc headers do not select the
+tenant. Retries for the same raw message and exact SMTP recipient are deduplicated.
+
+Inside your authenticated application's mailbox view/service:
+
+```ruby
+inboxes.for_tenant("organization-123") do |account|
+  mailbox = account.mailboxes.find_by!(owner_ref: "team:42")
+  account.messages(mailbox.id).inbox.unread.order(id: :desc).limit(25).each do |entry|
+    inbound = account.inbound_email(mailbox.id, entry.id)
+    subject = inbound.mail.subject
+    raw_mime = inbound.raw_email.download
+    # Build your authorized UI response here; sanitize rendered mail content.
+  end
+end
+```
+
+Perform tenant record access inside the block, rather than retaining lazy
+relations or framework records for use under another tenant. Numeric record IDs
+can overlap in separate tenant databases.
+
+For a message entry you have already authorized:
+
+```ruby
+account.mark_read(mailbox.id, entry.id)
+account.mark_read(mailbox.id, entry.id, read: false)
+account.archive(mailbox.id, entry.id)
+account.archive(mailbox.id, entry.id, archived: false)
+```
+
+These calls also belong inside the tenant block. Raw mail with a mailbox
+membership is protected from ActionMailbox's normal automatic incineration.
+Archive hides it from the inbox scope but retains the content. For deliberate
+permanent deletion, use `account.purge_message(mailbox.id, entry.id)`; it deletes
+the raw inbound record only when no other mailbox membership references it.
+ActiveStorage schedules its attachment cleanup jobs. Apply your own retention,
+encryption and backup policies; archive is not a retention policy.
+
+## Send from the mailbox
+
+Use a rendered ActionMailer message whose From and SMTP envelope sender are an
+active address belonging to the mailbox. The domain must be enabled for sending.
+`ReplyMailer` below is your application's mailer, not a generated gem class.
+
+```ruby
+inboxes.for_tenant("organization-123") do |account|
+  mailbox = account.mailboxes.find_by!(owner_ref: "team:42")
+  operation = account.prepare(mailbox.id,
+    operation_key: "draft:789:revision:1",
+    mail: ReplyMailer.reply(draft).message,
+  )
+  account.enqueue(mailbox.id, operation_key: operation.operation_key)
+end
+```
+
+Authorize the send before preparation. If preparation is part of an application
+transaction, enqueue only after its **outer commit**. The job queue must process
+`mailers`. Do not also call `deliver_later` on the mailer.
+
+The caller's operation key is namespaced by tenant and mailbox. Save the returned
+operation key and reuse it when enqueueing a retry. Re-rendering an already
+prepared operation may change MIME headers and raises a snapshot conflict.
+The job serializes tenant, mailbox ID and operation key—not MIME or credentials.
+It checks the current mailbox/address/domain state again before sending.
+
+Inspect `account.outbound_messages(mailbox.id)` and each link's
+`outbound_delivery` for the saved operation, provider ID and recipient outcomes.
+Accepted or partial attempts are never automatically resent. Uncertain attempts
+need [audited reconciliation](outbox.md#reconcile-uncertainty), available through
+`account.reconcile(mailbox.id, operation_key: saved_key, **evidence)`.
+
+For several Cloudflare accounts, configure a runtime client resolver **before
+loading mailbox models**:
+
+```ruby
+Cloudflare::Email::Mailboxes.configure(
+  directory_base: DirectoryRecord,
+  client_resolver: ->(tenant_key, account_id) {
+    Cloudflare::Email::Client.new(account_id: account_id,
+      api_token: YourSecretStore.email_token(tenant_key, account_id),
+      retry_ambiguous: false)
+  },
+)
+```
+
+`DirectoryRecord` and `YourSecretStore` are application examples. For one account,
+the default resolver uses the existing gem credentials and verifies the account
+matches. Cloudflare account IDs and your tenant keys are separate identities.
+
+## Delivery events and recovery
+
+Configure the [Cloudflare Queue subscription and token](delivery-events.md).
+Use shared intake for tenant mailboxes, rather than selecting a tenant from the
+event's recipient or Cloudflare account:
+
+```ruby
+Rails.application.config.x.cloudflare_email.event_domains = ["acme.example.com"]
+Rails.application.config.x.cloudflare_email.event_handler = ->(event) {
+  Cloudflare::Email::Mailboxes::Events.record(event)
+}
+```
+
+Schedule `cloudflare:email:consume_events` to poll batches. Also schedule these
+jobs regularly, starting at the beginning of each scan:
+
+```ruby
+Cloudflare::Email::Mailboxes::ReplayEventsJob.perform_later
+Cloudflare::Email::Mailboxes::RecoverJob.perform_later("organization-123")
+```
+
+Schedule recovery for each provisioned organization in your own directory. Each
+job processes a bounded page and enqueues its continuation. Recovery dispatches
+prepared operations, repairs accepted-result callbacks and registers provider
+correlation; it does not clear uncertain send claims. Domain suspension leaves
+events retained for later projection.
+
+Shared event intake commits before queue acknowledgement. After an accepted
+send, the gem registers account/message/recipient correlation to the tenant's
+outbox record. Events arriving first remain unmatched until replay. Tenant
+projection commits before shared completion; a crash between those commits
+causes safe redelivery to the tenant's deduplicating receipt ledger. Ambiguous
+correlations remain unprojected.
+
+The existing `config.x.cloudflare_email.outbox_delivery_handler` and
+`outbox_recipient_handler` callbacks work here too. Keep them idempotent and on
+the same tenant connection when atomicity is required. External effects cannot
+be rolled back. Provider acceptance, shared storage and tenant storage are not
+one transaction; this is not exactly-once delivery.
+
+## Isolation and rollout
+
+The scoped session API checks tenant/mailbox ownership and sender addresses.
+It does not authenticate Rails users. Protect admin directory APIs, choose the
+tenant through your access resolver, and authorize every mailbox operation.
+Direct/unscoped ActiveRecord access is an application-level privileged interface,
+particularly in shared-database mode; it is not a row-level authorization system.
+
+For database tenancy, configure all framework storage on the tenant connection
+and exclude ingress from hostname/session tenant selection. New framework jobs
+capture tenant context before serialization and restore it before GlobalID
+lookup. Drain existing ActionMailbox/ActiveStorage queues before enabling this
+mode: previously serialized jobs without the new metadata are rejected.
+
+The gem does not create organizations/users, verify customer domain ownership,
+provide IMAP/POP, or supply a full compose/conversation inbox. The optional
+[management engine](management-engine.md) supplies mailbox administration and
+plain-text previews using your host's access policy. The original Rebulk document-processing
+Worker contract and sender-review policy still need a separate application
+migration; installing this module does not replace that live pipeline.

data/docs/management-engine.md

@@ -0,0 +1,190 @@
+# Optional mailbox management engine
+
+The management engine is a small, server-rendered Rails interface for the
+[managed mailbox APIs](mailboxes.md). It works with ordinary Rails apps and with
+apps using Inertia, React, Turbo or another frontend. No JavaScript, Node build,
+asset pipeline or frontend package is required for the engine.
+
+This feature is available in RubyGems 0.3.0.
+
+## What you get
+
+- Mailbox lists, names, creation and aliases on allowed receiving domains.
+- Address setup status and mailbox suspension/resumption.
+- A paginated list of received messages and an escaped plain-text preview.
+- Optional read/unread and archive/unarchive controls.
+- Ordinary CSRF-protected forms, mobile layouts and bundled CSS.
+
+Your app supplies login, permissions, domain entitlements and any organization
+mapping. The engine does not create users, claim domains, edit DNS, provision
+Cloudflare routes, send mail, or permanently delete records. The default creation
+hooks create pending addresses. Your administrator or existing provisioning code
+completes routing and activation through the same Ruby APIs.
+
+The viewer never renders sender HTML, loads remote images, or provides raw MIME
+or attachment downloads. It shows attachment names and up to 100,000 plain-text
+characters. Compose, conversations, AI workflows and product search belong in
+your application.
+
+## Install and mount
+
+First install the managed mailbox tables and enable the module using
+[the mailbox setup guide](mailboxes.md). SQLite works; database tenancy remains
+off unless your app configures it explicitly.
+
+Require the engine **before Rails initializes**, after Bundler loads the gem:
+
+```ruby
+# config/application.rb, after Bundler.require(*Rails.groups)
+require "cloudflare/email/management"
+```
+
+Configure a host adapter in an initializer:
+
+```ruby
+# config/initializers/cloudflare_email_management.rb
+Cloudflare::Email::Management.configure do |config|
+  # Resolve the class when handling the request, so Rails reloading works.
+  config.adapter = ->(controller) { EmailManagementAccess.new(controller) }
+  config.back_path = ->(controller) { controller.main_app.root_path }
+end
+```
+
+Mount it wherever it fits your application:
+
+```ruby
+# config/routes.rb
+mount Cloudflare::Email::Management::Engine => "/email-management",
+  as: :email_management
+```
+
+Installing the gem does not expose this interface automatically. Without a
+configured adapter and enabled mailbox module the mounted interface returns 503.
+The base adapter grants no access. Its public stylesheet endpoint serves only
+the bundled CSS.
+
+## Connect your authentication and ownership
+
+Here is a single-database example using Rails' generated authentication convention
+(`Session`, `Current.session`, and a signed `session_id` cookie). Match the cookie
+and session lookup to your app; the agentic inbox example uses `session_token`.
+Use the host authentication system's expiry, revocation and access policy.
+
+```ruby
+# app/services/email_management_access.rb
+class EmailManagementAccess < Cloudflare::Email::Management::Adapter
+  ACTIONS = %i[index show create add_address suspend resume
+    show_message mark_read archive].freeze
+
+  def authenticate!
+    Current.session = Session.find_by(
+      id: controller.request.cookie_jar.signed[:session_id]
+    )
+    @user = Current.user
+    return true if @user
+
+    controller.redirect_to controller.main_app.new_session_path
+    false
+  end
+
+  def tenant_key = "application"
+  def owner_ref = "User:#{@user.id}"
+
+  def mailboxes(session)
+    session.mailboxes.where(owner_ref: owner_ref)
+  end
+
+  def allowed?(action, mailbox = nil)
+    @user.present? && ACTIONS.include?(action) &&
+      (mailbox.nil? || mailbox.owner_ref == owner_ref)
+  end
+
+  def domains(_session)
+    ["in.example.com"] # Only domains this user is entitled to use.
+  end
+
+  def create_mailbox(session, name:, address:)
+    session.create(name: name, address: address, owner_ref: owner_ref)
+  end
+end
+```
+
+For Devise, the authentication hook can use your configured Warden scope instead
+of the cookie lookup, for example
+`@user = controller.request.env["warden"]&.authenticate(scope: :user)`.
+The engine inherits from `ActionController::Base`; host `ApplicationController`
+callbacks and methods are not inherited. Make authentication explicit in the
+adapter and return exactly `true` on success, or redirect to the host's login.
+
+The directory must already contain the active receiving domain assigned to the
+same registry key. The engine intersects `domains` with that directory and
+intersects the host mailbox relation with the current gem tenant. Parameters
+cannot select an owner or tenant.
+
+Do not return every domain merely because it exists in the database. A customer
+app should apply organization entitlement, reserved-name and signup policies.
+For separate tenant databases, return a trusted host-resolved key from
+`tenant_key` and configure the existing gem tenancy adapter and framework
+connections before mailbox models load.
+
+## Customize behavior without forking the pages
+
+The adapter has these hooks:
+
+| Hook | Contract |
+| --- | --- |
+| `authenticate!` | Authenticate the request; return exactly true or redirect/deny. |
+| `tenant_key` | Trusted registry key; never use an unchecked request parameter. |
+| `mailboxes(session)` | ActiveRecord relation containing mailboxes accessible to this principal. |
+| `allowed?(action, mailbox = nil)` | Return exactly true to permit an action. |
+| `domains(session)` | Array of domain strings permitted for this principal. |
+| `create_mailbox(session, name:, address:)` | Return the created gem mailbox, visible in the host's scope. |
+| `add_address(session, mailbox, address:)` | Add an address through your existing app provisioning service if needed. |
+
+The adapter factory, authentication and tenant-key resolution run before the
+mailbox context is entered; the host must establish any context its authentication
+needs. Mailbox/domain scopes, permissions and mutation hooks run inside the
+resolved mailbox context. Creation also runs in a
+database transaction; a returned mailbox outside the host scope rolls back.
+Keep network work out of creation hooks and enqueue provisioning after commit.
+The default alias hook calls `session.add_address`.
+
+Return false for `:create` or lifecycle actions to provide a restricted dashboard.
+Use `:show_message` to control message access independently of metadata. Deny
+`:mark_read` and `:archive` when the host product already owns a different
+conversation-level read/folder model. Hidden controls also reject direct requests.
+
+An application with its own mailbox record can override creation and alias hooks
+to call that application's service and return the linked gem model. The agentic
+inbox example does this: both UIs and automatic account provisioning share one
+service, so engine-created mailboxes also appear in its full inbox.
+
+## Inertia, React and Turbo applications
+
+Use a normal HTML link or full-page navigation to the mount. For example:
+
+```jsx
+<a href="/email-management">Manage mailboxes</a>
+```
+
+The engine returns HTML, not Inertia responses. Its separate layout keeps host
+frontend dependencies out of the gem. The optional `back_path` callback returns
+a local absolute path for a Back to app link. Programmatic provisioning continues
+to work whether this engine is mounted or not.
+
+## Operational behavior
+
+Suspension preserves mailbox records and retained mail while disabling mailbox
+sending/receiving. Resuming does not activate pending addresses or repair DNS.
+Read/archive state belongs to the gem mailbox memberships and does not implicitly
+rewrite host conversation records.
+
+Private pages use no-store caching, a restrictive CSP and a same-origin referrer
+policy. Forms check both Rails CSRF tokens and origins. Serve the host over HTTPS,
+configure trusted proxy headers correctly, and keep its session cookies secure.
+Do not disable CSRF to work around a proxy configuration issue.
+
+Verification includes real Rails sessions, owner/tenant scoping, all operation
+permissions, CSRF/origin checks, rollback, escaped previews, retained raw source,
+and eager-loading a core-only app without exposing the engine. The example also
+tests desktop/mobile Chromium and JavaScript-disabled forms with synthetic mail.