telex 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 21555637ece74cdb455b24dddc9dcfd7193b8ec0a2cad21d0da1052a583d778c
+  data.tar.gz: e0f91a8b8acfa3f81db1b51bf13cc2bf8af522cc83106676b8c52563ae30b2f2
+SHA512:
+  metadata.gz: 852c7dd0cbe3fc1011a99bbf0a98dc54c398390f66717c0a08143626e6c798aa41866d0b11e86618652f18ef26d7d59ade28a4b17df5591f4ad92fb37b396e1a
+  data.tar.gz: c563df784e2e03f8ea8521d3566d4f677967bac6ee762f7e9d33497f6825faedc5e9c19a22c56c987fb765110c338a2664e7f61f6defab22975d9c7b6f17766a

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright B.O.X
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,101 @@
+# Telex
+
+Email for our Rails apps through [Lettermint](https://lettermint.co), a Dutch provider with servers in the EU.
+
+- **Sending:** a `:lettermint` delivery method for Action Mailer, over Lettermint's SMTP relay. Mailers, previews, `deliver_later` and gems that send mail (Devise…) work as usual.
+- **Receiving:** a Lettermint ingress for Action Mailbox, next to the ones Rails ships for Postmark or Mailgun. Lettermint posts a signed webhook with a link to the original message; Telex checks the signature, downloads the message in a job and hands it to Action Mailbox, which routes it to the app's mailboxes.
+
+Each app has its own Lettermint project: its own API token, logs, webhook and receiving subdomain (`reply.hq.box.paris`, `reply.ticket.box.paris`…).
+
+## Install in an app
+
+```ruby
+# Gemfile
+gem "telex", github: "boxprod/telex"
+```
+
+```yaml
+# bin/rails credentials:edit
+lettermint:
+  api_token: lm_...        # the project's API token, also the SMTP password
+  webhook_secret: whsec_... # the inbound route's webhook secret, whsec_ included
+```
+
+Or `LETTERMINT_API_TOKEN` and `LETTERMINT_WEBHOOK_SECRET` in the environment.
+
+### Sending
+
+```ruby
+# config/environments/production.rb
+config.action_mailer.delivery_method = :lettermint
+```
+
+The sending domain must be verified in the Lettermint project (SPF, DKIM, return path). Lettermint's [SMTP headers](https://lettermint.co/docs/guides/send-email-with-smtp) (`X-Lettermint-Route`, `X-LM-Tag`, `X-LM-Metadata-*`…) can be set in a mailer with `headers[...]`.
+
+### Receiving
+
+Install Action Mailbox if the app does not have it yet (it needs Active Storage):
+
+```sh
+bin/rails action_mailbox:install db:migrate
+```
+
+```ruby
+# config/environments/production.rb
+config.action_mailbox.ingress = :lettermint
+```
+
+In the Lettermint project, add an inbound route on the receiving subdomain (its MX records point to Lettermint) with the webhook
+
+```
+https://<the app>/rails/action_mailbox/lettermint/inbound_emails
+```
+
+Then route messages in `app/mailboxes/application_mailbox.rb` as with any Action Mailbox app. Inbound needs Lettermint's Starter plan or above.
+
+What Telex adds to each message before Action Mailbox sees it:
+
+| Header | |
+|---|---|
+| `X-Original-To` | The address it was delivered to, which Action Mailbox routes on even when it is not in To or Cc |
+| `X-Lettermint-Spam` | `yes` or `no`, Lettermint's verdict; nothing is dropped |
+| `X-Lettermint-Spam-Score` | Lettermint's score |
+
+### How it behaves
+
+- A wrong or missing signature, or one more than 5 minutes old, gets a `401`. Other events (deliveries, bounces) get a `204` and are ignored for now.
+- The webhook answers once the download is queued. If the app cannot queue it, Lettermint gets a `500` and tries again.
+- The job tries again for several hours when Lettermint cannot be reached, and gives up at once if the link has expired (28 days after the message arrived).
+- A message that arrives twice is kept once: Action Mailbox ignores a source it already has.
+- The links are secrets: the webhook's `data` is filtered from the request log and the job does not log its arguments.
+
+## Testing in an app
+
+To test a mailbox, Action Mailbox's own `receive_inbound_email_from_mail` is the simplest. To test the webhook end to end:
+
+```ruby
+require "telex/test_helper"
+
+class InboundTest < ActionDispatch::IntegrationTest
+  include Telex::TestHelper
+
+  test "a reply reaches its quote" do
+    body = lettermint_inbound_payload(recipient: "reply+quote-12@reply.hq.box.paris")
+    post rails_lettermint_inbound_emails_path, params: body, headers: lettermint_webhook_headers(body)
+    # Telex.download = an object answering #fetch(url) with the message's source, to run the job without the network
+  end
+end
+```
+
+## Not done yet
+
+- Delivery events (bounces, complaints) are ignored. They would come through the same webhook.
+- Whether Lettermint keeps the Message-ID Rails generates, which replies point to in `In-Reply-To`: Lettermint has an `X-LM-Preserve-Message-ID` header for it, to check against a real reply before relying on threading.
+
+## Development
+
+```sh
+bundle install
+bin/rails db:migrate
+bin/rails test
+```

data/Rakefile

@@ -0,0 +1,6 @@
+require "bundler/setup"
+
+APP_RAKEFILE = File.expand_path("test/dummy/Rakefile", __dir__)
+load "rails/tasks/engine.rake"
+
+require "bundler/gem_tasks"

data/app/controllers/action_mailbox/ingresses/lettermint/inbound_emails_controller.rb

@@ -0,0 +1,51 @@
+module ActionMailbox
+  # Ingests inbound emails from Lettermint, whose webhooks carry a signed link to the original
+  # message rather than the message itself. The link is fetched by Telex::IngestJob.
+  #
+  # Authenticates requests by their X-Lettermint-Signature header.
+  #
+  # Returns:
+  #
+  # - <tt>204 No Content</tt> once the message is queued for download, and for every other event
+  # - <tt>401 Unauthorized</tt> if the signature is wrong or more than 5 minutes old
+  # - <tt>404 Not Found</tt> if Action Mailbox is not configured to accept inbound emails from Lettermint
+  # - <tt>422 Unprocessable Entity</tt> if an inbound event has no link to the message
+  # - <tt>500 Server Error</tt> if the webhook secret is missing, or the Active Job backend is unavailable
+  class Ingresses::Lettermint::InboundEmailsController < ActionMailbox::BaseController
+    before_action :authenticate
+
+    def create
+      if payload["event"] == "message.inbound"
+        data = payload.fetch("data", {})
+        url = data.dig("raw", "url")
+        return head :unprocessable_entity if url.blank?
+
+        Telex::IngestJob.perform_later(url, recipient: data["recipient"], spam_score: data["spam_score"], spam: data["is_spam"])
+      end
+
+      head :no_content
+    end
+
+    private
+      # The payload holds the message, and links Lettermint asks to keep secret: none of it goes to the logs.
+      def process_action(...)
+        request.set_header "action_dispatch.parameter_filter", Rails.application.config.filter_parameters + [ /\Adata\z/ ]
+        super
+      end
+
+      def payload
+        @payload ||= JSON.parse(request.raw_post)
+      end
+
+      def authenticate
+        head :unauthorized unless Telex::Signature.new(request.headers["X-Lettermint-Signature"], secret: secret).valid?(request.raw_post)
+      end
+
+      def secret
+        Telex.webhook_secret.presence || raise(ArgumentError, <<~MESSAGE.squish)
+          Missing the Lettermint webhook secret. Set lettermint.webhook_secret in your application's
+          encrypted credentials or provide the LETTERMINT_WEBHOOK_SECRET environment variable.
+        MESSAGE
+      end
+  end
+end

data/app/jobs/telex/ingest_job.rb

@@ -0,0 +1,34 @@
+module Telex
+  # Downloads an inbound message and hands it to Action Mailbox. Lettermint retries webhooks,
+  # and Action Mailbox ignores a message it already has, so running twice is harmless.
+  #
+  # Not the host's ApplicationJob: none of its callbacks or retries apply here.
+  class IngestJob < ActiveJob::Base
+    queue_as :default
+
+    # The first argument is the signed link to the message.
+    self.log_arguments = false
+
+    retry_on Download::Error, wait: :polynomially_longer, attempts: 10
+
+    discard_on Download::Gone do |job, error|
+      Rails.logger.error "[Telex] Inbound email dropped: #{error.message} (job #{job.job_id})"
+    end
+
+    def perform(url, recipient: nil, spam_score: nil, spam: nil)
+      source = Telex.download.fetch(url)
+      ActionMailbox::InboundEmail.create_and_extract_message_id! headers(recipient, spam_score, spam) + source
+    end
+
+    private
+      # The address the message was delivered to, which may not be in its To or Cc (a Bcc, a list),
+      # is what Action Mailbox routes on. Lettermint's spam verdict comes along for mailboxes to use.
+      def headers(recipient, spam_score, spam)
+        lines = []
+        lines << "X-Original-To: #{recipient}" if recipient.present?
+        lines << "X-Lettermint-Spam: #{spam ? "yes" : "no"}" unless spam.nil?
+        lines << "X-Lettermint-Spam-Score: #{spam_score}" unless spam_score.nil?
+        lines.map { |line| "#{line}\r\n" }.join.b
+      end
+  end
+end

data/config/routes.rb

@@ -0,0 +1,5 @@
+Rails.application.routes.draw do
+  scope "/rails/action_mailbox", module: "action_mailbox/ingresses" do
+    post "/lettermint/inbound_emails" => "lettermint/inbound_emails#create", as: :rails_lettermint_inbound_emails
+  end
+end

data/lib/telex/download.rb

@@ -0,0 +1,41 @@
+require "net/http"
+
+module Telex
+  # Fetches a message's original source from the signed URL in the webhook. The URL needs no
+  # credentials, so it is a secret until it expires, 28 days after the message arrived.
+  class Download
+    # Worth trying again later.
+    class Error < StandardError; end
+
+    # The link has expired or was never valid: trying again will not help.
+    class Gone < Error; end
+
+    MAX_REDIRECTS = 3
+
+    def fetch(url, redirects: MAX_REDIRECTS)
+      uri = URI(url)
+      raise Gone, "Lettermint gave an address that is not HTTPS" unless uri.is_a?(URI::HTTPS)
+
+      case response = perform(uri)
+      when Net::HTTPSuccess
+        response.body.b
+      when Net::HTTPRedirection
+        raise Error, "Lettermint redirected too many times" if redirects.zero?
+        fetch(URI.join(uri, response["location"]).to_s, redirects: redirects - 1)
+      when Net::HTTPForbidden, Net::HTTPNotFound, Net::HTTPGone
+        raise Gone, "Lettermint answered #{response.code} for the message's source"
+      else
+        raise Error, "Lettermint answered #{response.code} for the message's source"
+      end
+    rescue Timeout::Error, SystemCallError, SocketError, OpenSSL::SSL::SSLError, Net::HTTPBadResponse => error
+      raise Error, "Lettermint could not be reached (#{error.class})"
+    end
+
+    private
+      def perform(uri)
+        Net::HTTP.start(uri.host, uri.port, use_ssl: true, open_timeout: 10, read_timeout: 60) do |http|
+          http.request(Net::HTTP::Get.new(uri))
+        end
+      end
+  end
+end

data/lib/telex/engine.rb

@@ -0,0 +1,12 @@
+module Telex
+  # Not isolated: the ingress lives in Action Mailbox's namespace and its route in the app's,
+  # next to the ones Rails ships for other providers.
+  class Engine < ::Rails::Engine
+    # config.action_mailer.delivery_method = :lettermint
+    initializer "telex.delivery_method" do
+      ActiveSupport.on_load(:action_mailer) do
+        add_delivery_method :lettermint, Mail::SMTP, Telex::SMTP_SETTINGS.merge(password: Telex.api_token)
+      end
+    end
+  end
+end

data/lib/telex/signature.rb

@@ -0,0 +1,31 @@
+require "openssl"
+
+module Telex
+  # The X-Lettermint-Signature header: "t=<unix seconds>,v1=<hex HMAC-SHA256 of "<t>.<raw body>">",
+  # keyed with the webhook secret as given, whsec_ prefix included.
+  class Signature
+    TOLERANCE = 5.minutes
+
+    def self.generate(body, secret:, at: Time.now)
+      timestamp = at.to_i
+      "t=#{timestamp},v1=#{digest(timestamp, body, secret)}"
+    end
+
+    def self.digest(timestamp, body, secret)
+      OpenSSL::HMAC.hexdigest("SHA256", secret, "#{timestamp}.#{body}")
+    end
+
+    def initialize(header, secret:)
+      @parts = header.to_s.split(",").to_h { |part| part.split("=", 2).map(&:strip) }
+      @secret = secret
+    end
+
+    def valid?(body, now: Time.now)
+      timestamp = Integer(@parts["t"], exception: false)
+      return false unless timestamp && @parts["v1"]
+
+      (now.to_i - timestamp).abs <= TOLERANCE &&
+        ActiveSupport::SecurityUtils.secure_compare(@parts["v1"], self.class.digest(timestamp, body, @secret))
+    end
+  end
+end

data/lib/telex/test_helper.rb

@@ -0,0 +1,24 @@
+module Telex
+  # Signs requests the way Lettermint does, for integration tests of the inbound webhook.
+  # To test a mailbox, Action Mailbox's own receive_inbound_email_from_mail is simpler.
+  module TestHelper
+    def lettermint_webhook_headers(body, secret: Telex.webhook_secret, event: JSON.parse(body)["event"], at: Time.now)
+      {
+        "Content-Type" => "application/json",
+        "X-Lettermint-Signature" => Telex::Signature.generate(body, secret: secret, at: at),
+        "X-Lettermint-Event" => event,
+        "X-Lettermint-Delivery" => at.to_i.to_s,
+        "X-Lettermint-Attempt" => "1"
+      }
+    end
+
+    def lettermint_inbound_payload(recipient:, raw_url: "https://storage.lettermint.co/inbound/raw/test?signature=test", **data)
+      {
+        id: SecureRandom.uuid,
+        event: "message.inbound",
+        timestamp: Time.now.utc.iso8601(3),
+        data: { recipient: recipient, raw: { url: raw_url, expires_at: 28.days.from_now.iso8601 }, is_spam: false, spam_score: 0.0, **data }
+      }.to_json
+    end
+  end
+end

data/lib/telex/version.rb

@@ -0,0 +1,3 @@
+module Telex
+  VERSION = "0.1.0"
+end

data/lib/telex.rb

@@ -0,0 +1,35 @@
+require "telex/version"
+require "telex/signature"
+require "telex/download"
+require "telex/engine"
+
+# Email for our Rails apps through Lettermint: Action Mailer sends over its SMTP relay,
+# Action Mailbox receives from its inbound webhooks.
+module Telex
+  SMTP_SETTINGS = {
+    address: "smtp.lettermint.co",
+    port: 587,
+    user_name: "lettermint",
+    authentication: :plain,
+    enable_starttls: true
+  }.freeze
+
+  class << self
+    # The project's API token, which is also the SMTP password.
+    def api_token
+      Rails.application.credentials.dig(:lettermint, :api_token) || ENV["LETTERMINT_API_TOKEN"]
+    end
+
+    # The inbound route's webhook secret, whsec_ prefix included.
+    def webhook_secret
+      Rails.application.credentials.dig(:lettermint, :webhook_secret) || ENV["LETTERMINT_WEBHOOK_SECRET"]
+    end
+
+    # Replaced in tests; anything that answers #fetch(url) with the message's source.
+    attr_writer :download
+
+    def download
+      @download ||= Download.new
+    end
+  end
+end

metadata

@@ -0,0 +1,67 @@
+--- !ruby/object:Gem::Specification
+name: telex
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- B.O.X
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rails
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '8.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '8.0'
+description: Action Mailer sends through Lettermint's SMTP relay; Action Mailbox receives
+  from Lettermint's inbound webhooks.
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- MIT-LICENSE
+- README.md
+- Rakefile
+- app/controllers/action_mailbox/ingresses/lettermint/inbound_emails_controller.rb
+- app/jobs/telex/ingest_job.rb
+- config/routes.rb
+- lib/telex.rb
+- lib/telex/download.rb
+- lib/telex/engine.rb
+- lib/telex/signature.rb
+- lib/telex/test_helper.rb
+- lib/telex/version.rb
+homepage: https://github.com/boxprod/telex
+licenses:
+- MIT
+metadata:
+  rubygems_mfa_required: 'true'
+  source_code_uri: https://github.com/boxprod/telex
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '3.3'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 4.0.20
+specification_version: 4
+summary: Email for our Rails apps through Lettermint.
+test_files: []