mail_room 0.10.1 → 0.11.1

data/README.md

@@ -111,10 +111,23 @@ You will also need to install `faraday` or `letter_opener` if you use the `postb
       :client_id: ABCDE
       :client_secret: YOUR-SECRET-HERE
       :poll_interval: 60
+      :azure_ad_endpoint: https://login.microsoftonline.com
+      :graph_endpoint: https://graph.microsoft.com
     :delivery_method: sidekiq
     :delivery_options:
       :redis_url: redis://localhost:6379
       :worker: EmailReceiverWorker
+  -
+    :email: "user8@gmail.com"
+    :password: "password"
+    :name: "inbox"
+    :delivery_method: postback
+    :delivery_options:
+      :delivery_url: "http://localhost:3000/inbox"
+      :jwt_auth_header: "Mailroom-Api-Request"
+      :jwt_issuer: "mailroom"
+      :jwt_algorithm: "HS256"
+      :jwt_secret_path: "/etc/secrets/mailroom/.mailroom_secret"
 ```
 
 **Note:** If using `delete_after_delivery`, you also probably want to use
@@ -135,11 +148,11 @@ If you're seeing the error `Please log in via your web browser: https://support.
 ### Microsoft Graph configuration
 
 To use the Microsoft Graph API instead of IMAP to read e-mail, you will
-need to create an application in the Azure Active Directory. See the
-[Microsoft instructions](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app) for more details:
+need to create an application in Microsoft Entra ID (formerly known as Azure Active Directory). See the
+[Microsoft instructions](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app) for more details:
 
 1. Sign in to the [Azure portal](https://portal.azure.com).
-1. Search for and select `Azure Active Directory`.
+1. Search for and select `Microsoft Entra ID`.
 1. Under `Manage`, select `App registrations` > `New registration`.
 1. Enter a `Name` for your application, such as `MailRoom`. Users of your app might see this name, and you can change it later.
 1. If `Supported account types` is listed, select the appropriate option.
@@ -186,6 +199,25 @@ Fill in `inbox_options` with these values:
 By default, MailRoom will poll for new messages every 60 seconds. `poll_interval` configures the number of
 seconds to poll. Setting the value to 0 or under will default to 60 seconds.
 
+### Alternative Azure cloud deployments
+
+MailRoom will default to using the standard Azure HTTPS endpoints. To
+configure MailRoom with Microsoft Cloud for US Government or other
+[national cloud deployments](https://docs.microsoft.com/en-us/graph/deployments), set
+the `azure_ad_endpoint` and `graph_endpoint` accordingly. For example,
+for Microsoft Cloud for US Government:
+
+```yaml
+    :inbox_method: microsoft_graph
+    :inbox_options:
+      :tenant_id: 12345
+      :client_id: ABCDE
+      :client_secret: YOUR-SECRET-HERE
+      :poll_interval: 60
+      :azure_ad_endpoint: https://login.microsoftonline.us
+      :graph_endpoint: https://graph.microsoft.us
+```
+
 ## delivery_method ##
 
 ### postback ###
@@ -215,7 +247,7 @@ Delivery options:
   Required, defaults to `redis://localhost:6379`.
 - **sentinels**: A list of sentinels servers used to provide HA to Redis. (see [Sentinel Support](#sentinel-support))
   Optional.
-- **namespace**: The Redis namespace Sidekiq works under. Use the same Redis namespace that's used to configure Sidekiq.
+- **namespace**: [DEPRECATED] The Redis namespace Sidekiq works under. Use the same Redis namespace that's used to configure Sidekiq.
   Optional.
 - **queue**: The Sidekiq queue the job is pushed onto. Make sure Sidekiq actually reads off this queue.
   Required, defaults to `default`.
@@ -312,8 +344,8 @@ And finally, configure MailRoom to use the postback configuration with the optio
 :delivery_method: postback
 :delivery_options:
   :delivery_url: https://example.com/rails/action_mailbox/relay/inbound_emails
-  :delivery_username: actionmailbox
-  :delivery_password: <INGRESS_PASSWORD>
+  :username: actionmailbox
+  :password: <INGRESS_PASSWORD>
 ```
 
 ## Receiving `postback` in Rails ##
@@ -359,7 +391,7 @@ When running multiple instances of MailRoom against a single mailbox, to try to
     :arbitration_options:
       # The Redis server to connect with. Defaults to redis://localhost:6379.
       :redis_url: redis://redis.example.com:6379
-      # The Redis namespace to house the Redis keys under. Optional.
+      # [DEPRECATED] The Redis namespace to house the Redis keys under. Optional.
       :namespace: mail_room
   -
     :email: "user2@gmail.com"
@@ -379,7 +411,7 @@ When running multiple instances of MailRoom against a single mailbox, to try to
         -
           :host: 127.0.0.1
           :port: 26379
-      # The Redis namespace to house the Redis keys under. Optional.
+      # [DEPRECATED] The Redis namespace to house the Redis keys under. Optional.
       :namespace: mail_room
 ```
 
@@ -422,13 +454,3 @@ respectively and MailRoom will log there.
 4. Push to the branch (`git push origin my-new-feature`)
 5. Create new Pull Request
 6. If accepted, ask for commit rights
-
-## TODO ##
-
-1. specs, this is just a (working) proof of concept √
-2. finish code for POSTing to callback with auth √
-3. accept mailbox configuration for one account directly on the commandline; or ask for it
-4. add example rails endpoint, with auth examples
-5. add example configs for upstart/init.d √
-6. log to stdout √
-7. add a development mode that opens in letter_opener by ryanb √

data/lib/mail_room/arbitration/redis.rb

@@ -9,6 +9,13 @@ module MailRoom
           namespace = mailbox.arbitration_options[:namespace]
           sentinels = mailbox.arbitration_options[:sentinels]
 
+          if namespace
+            warn <<~MSG
+              Redis namespaces are deprecated. This option will be ignored in future versions.
+              See https://github.com/sidekiq/sidekiq/issues/2586 for more details."
+            MSG
+          end
+
           super(redis_url, namespace, sentinels)
         end
       end
@@ -31,7 +38,7 @@ module MailRoom
         # Any subsequent failure in the instance which gets the lock will be dealt
         # with by the expiration, at which time another instance can pick up the
         # message and try again.
-        client.set(key, 1, {nx: true, ex: expiration})
+        client.set(key, 1, nx: true, ex: expiration)
       end
 
       private

data/lib/mail_room/connection.rb

@@ -6,16 +6,23 @@ module MailRoom
 
     def initialize(mailbox)
       @mailbox = mailbox
+      @stopped = false
     end
 
     def on_new_message(&block)
       @new_message_handler = block
     end
 
+    def stopped?
+      @stopped
+    end
+
     def wait
       raise NotImplementedError
     end
 
-    def quit; end
+    def quit
+      @stopped = true
+    end
   end
 end

data/lib/mail_room/delivery/postback.rb

@@ -1,11 +1,12 @@
 require 'faraday'
+require "mail_room/jwt"
 
 module MailRoom
   module Delivery
     # Postback Delivery method
     # @author Tony Pitale
     class Postback
-      Options = Struct.new(:url, :token, :username, :password, :logger, :content_type) do
+      Options = Struct.new(:url, :token, :username, :password, :logger, :content_type, :jwt) do
         def initialize(mailbox)
           url =
             mailbox.delivery_url ||
@@ -17,23 +18,44 @@ module MailRoom
             mailbox.delivery_options[:delivery_token] ||
             mailbox.delivery_options[:token]
 
-          username = mailbox.delivery_options[:username]
-          password = mailbox.delivery_options[:password]
+          jwt = initialize_jwt(mailbox.delivery_options)
+
+          username =
+            mailbox.delivery_options[:username] ||
+            mailbox.delivery_options[:delivery_username]
+          password =
+            mailbox.delivery_options[:password] ||
+            mailbox.delivery_options[:delivery_password]
 
           logger = mailbox.logger
 
           content_type = mailbox.delivery_options[:content_type]
 
-          super(url, token, username, password, logger, content_type)
+          super(url, token, username, password, logger, content_type, jwt)
         end
 
         def token_auth?
           !self[:token].nil?
         end
 
+        def jwt_auth?
+          self[:jwt].valid?
+        end
+
         def basic_auth?
           !self[:username].nil? && !self[:password].nil?
         end
+
+        private
+
+        def initialize_jwt(delivery_options)
+          ::MailRoom::JWT.new(
+            header: delivery_options[:jwt_auth_header],
+            secret_path: delivery_options[:jwt_secret_path],
+            algorithm: delivery_options[:jwt_algorithm],
+            issuer: delivery_options[:jwt_issuer]
+          )
+        end
       end
 
       # Build a new delivery, hold the delivery options
@@ -49,24 +71,62 @@ module MailRoom
         connection = Faraday.new
 
         if @delivery_options.token_auth?
-          connection.token_auth @delivery_options.token
+          config_token_auth(connection)
         elsif @delivery_options.basic_auth?
-          connection.basic_auth(
-            @delivery_options.username,
-            @delivery_options.password
-          )
+          config_basic_auth(connection)
         end
 
         connection.post do |request|
           request.url @delivery_options.url
           request.body = message
-          # request.options[:timeout] = 3
-          request.headers['Content-Type'] = @delivery_options.content_type unless @delivery_options.content_type.nil?
+          config_request_content_type(request)
+          config_request_jwt_auth(request)
         end
 
         @delivery_options.logger.info({ delivery_method: 'Postback', action: 'message pushed', url: @delivery_options.url })
         true
       end
+
+      private
+
+      def config_request_content_type(request)
+        return if @delivery_options.content_type.nil?
+
+        request.headers['Content-Type'] = @delivery_options.content_type
+      end
+
+      def config_request_jwt_auth(request)
+        return unless @delivery_options.jwt_auth?
+
+        request.headers[@delivery_options.jwt.header] = @delivery_options.jwt.token
+      end
+
+      def config_token_auth(connection)
+        # connection.token_auth was removed in Faraday v2 in favor of connection.request(:authorization, 'Token', token)
+        if defined?(connection.token_auth)
+          connection.token_auth @delivery_options.token
+        else
+          connection.request(
+            :authorization, 'Token',
+            @delivery_options.token
+          )
+        end
+      end
+
+      def config_basic_auth(connection)
+        if defined?(connection.basic_auth)
+          connection.basic_auth(
+            @delivery_options.username,
+            @delivery_options.password
+          )
+        else
+          connection.request(
+            :authorization, :basic,
+            @delivery_options.username,
+            @delivery_options.password
+          )
+        end
+      end
     end
   end
 end

data/lib/mail_room/delivery/que.rb

@@ -1,5 +1,6 @@
 require 'pg'
 require 'json'
+require 'charlock_holmes'
 
 module MailRoom
   module Delivery
@@ -34,7 +35,7 @@ module MailRoom
       # deliver the message by pushing it onto the configured Sidekiq queue
       # @param message [String] the email message as a string, RFC822 format
       def deliver(message)
-        queue_job(message)
+        queue_job(utf8_encode_message(message))
         @options.logger.info({ delivery_method: 'Que', action: 'message pushed' })
       end
 
@@ -58,6 +59,19 @@ module MailRoom
 
         connection.exec(sql, [options.priority, options.job_class, options.queue, JSON.dump(args)])
       end
+
+      def utf8_encode_message(message)
+        message = message.dup
+
+        message.force_encoding("UTF-8")
+        return message if message.valid_encoding?
+
+        detection = CharlockHolmes::EncodingDetector.detect(message)
+        return message unless detection && detection[:encoding]
+
+        # Convert non-UTF-8 body UTF-8 so it can be dumped as JSON.
+        CharlockHolmes::Converter.convert(message, detection[:encoding], 'UTF-8')
+      end
     end
   end
 end

data/lib/mail_room/delivery/sidekiq.rb

@@ -18,6 +18,13 @@ module MailRoom
           worker    = mailbox.delivery_options[:worker]
           logger = mailbox.logger
 
+          if namespace
+            warn <<~MSG
+              Redis namespaces are deprecated. This option will be ignored in future versions.
+              See https://github.com/sidekiq/sidekiq/issues/2586 for more details."
+            MSG
+          end
+
           super(redis_url, namespace, sentinels, queue, worker, logger, redis_db)
         end
       end

data/lib/mail_room/jwt.rb

@@ -0,0 +1,39 @@
+# frozen_string_literal: true
+
+require 'faraday'
+require 'securerandom'
+require 'jwt'
+require 'base64'
+
+module MailRoom
+  # Responsible for validating and generating JWT token
+  class JWT
+    DEFAULT_ISSUER = 'mailroom'
+    DEFAULT_ALGORITHM = 'HS256'
+
+    attr_reader :header, :secret_path, :issuer, :algorithm
+
+    def initialize(header:, secret_path:, issuer:, algorithm:)
+      @header = header
+      @secret_path = secret_path
+      @issuer = issuer || DEFAULT_ISSUER
+      @algorithm = algorithm || DEFAULT_ALGORITHM
+    end
+
+    def valid?
+      [@header, @secret_path, @issuer, @algorithm].none?(&:nil?)
+    end
+
+    def token
+      return nil unless valid?
+
+      secret = Base64.strict_decode64(File.read(@secret_path).chomp)
+      payload = {
+        nonce: SecureRandom.hex(12),
+        iat: Time.now.to_i, # https://github.com/jwt/ruby-jwt#issued-at-claim
+        iss: @issuer
+      }
+      ::JWT.encode payload, secret, @algorithm
+    end
+  end
+end

data/lib/mail_room/microsoft_graph/connection.rb

@@ -6,7 +6,6 @@ require 'oauth2'
 module MailRoom
   module MicrosoftGraph
     class Connection < MailRoom::Connection
-      SCOPE = 'https://graph.microsoft.com/.default'
       NEXT_PAGE_KEY = '@odata.nextLink'
       DEFAULT_POLL_INTERVAL_S = 60
 
@@ -22,6 +21,8 @@ module MailRoom
       end
 
       def wait
+        return if stopped?
+
         process_mailbox
 
         @throttled_count = 0
@@ -42,17 +43,30 @@ module MailRoom
       private
 
       def wait_for_new_messages
-        sleep poll_interval
+        sleep_while_running(poll_interval)
       end
 
       def backoff
-        sleep backoff_secs
+        sleep_while_running(backoff_secs)
       end
 
       def backoff_secs
         [60 * 10, 2**throttled_count].min
       end
 
+      # Unless wake up periodically, we won't notice that the thread was stopped
+      # if we sleep the entire interval.
+      def sleep_while_running(sleep_interval)
+        sleep_interval.times do
+          do_sleep(1)
+          return if stopped?
+        end
+      end
+
+      def do_sleep(interval)
+        sleep(interval)
+      end
+
       def reset
         @token = nil
         @throttled_count = 0
@@ -61,12 +75,12 @@ module MailRoom
       def setup
         @mailbox.logger.info({ context: @mailbox.context, action: 'Retrieving OAuth2 token...' })
 
-        @token = client.client_credentials.get_token({ scope: SCOPE })
+        @token = client.client_credentials.get_token({ scope: scope })
       end
 
       def client
         @client ||= OAuth2::Client.new(client_id, client_secret,
-                                       site: 'https://login.microsoftonline.com',
+                                       site: azure_ad_endpoint,
                                        authorize_url: "/#{tenant_id}/oauth2/v2.0/authorize",
                                        token_url: "/#{tenant_id}/oauth2/v2.0/token",
                                        auth_scheme: :basic_auth)
@@ -192,7 +206,7 @@ module MailRoom
       end
 
       def base_url
-        "https://graph.microsoft.com/v1.0/users/#{mailbox.email}/mailFolders/#{mailbox.name}/messages"
+        "#{graph_endpoint}/v1.0/users/#{mailbox.email}/mailFolders/#{mailbox.name}/messages"
       end
 
       def unread_messages_url
@@ -201,7 +215,7 @@ module MailRoom
 
       def msg_url(id)
         # Attempting to use the base_url fails with "The OData request is not supported"
-        "https://graph.microsoft.com/v1.0/users/#{mailbox.email}/messages/#{id}"
+        "#{graph_endpoint}/v1.0/users/#{mailbox.email}/messages/#{id}"
       end
 
       def rfc822_msg_url(id)
@@ -212,6 +226,18 @@ module MailRoom
       def log_exception(message, exception)
         @mailbox.logger.warn({ context: @mailbox.context, message: message, exception: exception.to_s })
       end
+
+      def scope
+        "#{graph_endpoint}/.default"
+      end
+
+      def graph_endpoint
+        inbox_options[:graph_endpoint] || 'https://graph.microsoft.com'
+      end
+
+      def azure_ad_endpoint
+        inbox_options[:azure_ad_endpoint] || 'https://login.microsoftonline.com'
+      end
     end
   end
 end

data/lib/mail_room/version.rb

@@ -1,4 +1,4 @@
 module MailRoom
   # Current version of MailRoom gem
-  VERSION = "0.10.1"
+  VERSION = "0.11.1"
 end

data/mail_room.gemspec

@@ -18,19 +18,20 @@ Gem::Specification.new do |gem|
   gem.require_paths = ["lib"]
 
   gem.add_dependency             "net-imap", ">= 0.2.1"
-  gem.add_dependency             "oauth2", "~> 1.4.4"
+  gem.add_dependency             "oauth2", [">= 1.4.4", "< 3"]
+  gem.add_dependency             "jwt", ">= 2.0"
 
   gem.add_development_dependency "rake"
   gem.add_development_dependency "rspec", "~> 3.9"
   gem.add_development_dependency "rubocop", "~> 1.11"
-  gem.add_development_dependency "mocha", "~> 1.11"
+  gem.add_development_dependency "mocha", "~> 2.0"
   gem.add_development_dependency "simplecov"
 
   # for testing delivery methods
   gem.add_development_dependency "faraday"
   gem.add_development_dependency "mail"
   gem.add_development_dependency "letter_opener"
-  gem.add_development_dependency "redis", "~> 3.3.1"
+  gem.add_development_dependency "redis", "~> 4"
   gem.add_development_dependency "redis-namespace"
   gem.add_development_dependency "pg"
   gem.add_development_dependency "charlock_holmes"

data/spec/fixtures/jwt_secret

@@ -0,0 +1 @@
+aGVsbG93b3JsZA==

data/spec/lib/arbitration/redis_spec.rb

@@ -15,6 +15,7 @@ describe MailRoom::Arbitration::Redis do
 
   # Private, but we don't care.
   let(:redis) { subject.send(:client) }
+  let(:raw_client) { redis._client }
 
   describe '#deliver?' do
     context "when called the first time" do
@@ -95,7 +96,7 @@ describe MailRoom::Arbitration::Redis do
       it 'client has same specified url' do
         subject.deliver?(123)
 
-        expect(redis.client.options[:url]).to eq redis_url
+        expect(raw_client.options[:url]).to eq redis_url
       end
 
       it 'client is a instance of Redis class' do
@@ -137,10 +138,10 @@ describe MailRoom::Arbitration::Redis do
       before { ::Redis::Client::Connector::Sentinel.any_instance.stubs(:resolve).returns(sentinels) }
 
       it 'client has same specified sentinel params' do
-        expect(redis.client.instance_variable_get(:@connector)).to be_a Redis::Client::Connector::Sentinel
-        expect(redis.client.options[:host]).to eq('sentinel-master')
-        expect(redis.client.options[:password]).to eq('mypassword')
-        expect(redis.client.options[:sentinels]).to eq(sentinels)
+        expect(raw_client.instance_variable_get(:@connector)).to be_a Redis::Client::Connector::Sentinel
+        expect(raw_client.options[:host]).to eq('sentinel-master')
+        expect(raw_client.options[:password]).to eq('mypassword')
+        expect(raw_client.options[:sentinels]).to eq(sentinels)
       end
     end
   end

data/spec/lib/delivery/letter_opener_spec.rb

@@ -4,12 +4,13 @@ require 'mail_room/delivery/letter_opener'
 describe MailRoom::Delivery::LetterOpener do
   describe '#deliver' do
     let(:mailbox) {build_mailbox(location: '/tmp/somewhere')}
-    let(:delivery_method) {stub(:deliver!)}
+    let(:delivery_method) {stub}
     let(:mail) {stub}
 
     before :each do
       Mail.stubs(:read_from_string).returns(mail)
       ::LetterOpener::DeliveryMethod.stubs(:new).returns(delivery_method)
+      delivery_method.stubs(:deliver!)
     end
 
     it 'creates a new LetterOpener::DeliveryMethod' do

data/spec/lib/delivery/logger_spec.rb

@@ -19,7 +19,8 @@ describe MailRoom::Delivery::Logger do
       let(:mailbox) {build_mailbox(log_path: '/var/log/mail-room.log')}
 
       it 'creates a new file to append to' do
-        file = stub(:sync=)
+        file = stub
+        file.stubs(:sync=)
 
         File.expects(:open).with('/var/log/mail-room.log', 'a').returns(file)
         ::Logger.stubs(:new).with(file)

data/spec/lib/delivery/postback_spec.rb

@@ -65,11 +65,10 @@ describe MailRoom::Delivery::Postback do
           }
         })}
 
-  
         let(:delivery_options) {
           MailRoom::Delivery::Postback::Options.new(mailbox)
         }
-  
+
         it 'posts the message with faraday' do
           connection = stub
           request = stub
@@ -82,10 +81,59 @@ describe MailRoom::Delivery::Postback do
           connection.expects(:basic_auth).with('user1', 'password123abc')
 
           MailRoom::Delivery::Postback.new(delivery_options).deliver('a message')
-          
+
           expect(request.headers['Content-Type']).to eq('text/plain')
         end
       end
+
+      context 'with jwt token in the delivery options' do
+        let(:mailbox) {build_mailbox({
+          delivery_options: {
+            url: 'http://localhost/inbox',
+            jwt_auth_header: "Mailroom-Api-Request",
+            jwt_issuer: "mailroom",
+            jwt_algorithm: "HS256",
+            jwt_secret_path: "secret_path"
+          }
+        })}
+
+        let(:delivery_options) {
+          MailRoom::Delivery::Postback::Options.new(mailbox)
+        }
+
+        it 'posts the message with faraday' do
+          connection = stub
+          request = stub
+          Faraday.stubs(:new).returns(connection)
+
+          connection.expects(:post).yields(request).twice
+          request.stubs(:url)
+          request.stubs(:body=)
+          request.stubs(:headers).returns({})
+
+          jwt = stub
+          MailRoom::JWT.expects(:new).with(
+            header: 'Mailroom-Api-Request',
+            issuer: 'mailroom',
+            algorithm: 'HS256',
+            secret_path: 'secret_path'
+          ).returns(jwt)
+          jwt.stubs(:valid?).returns(true)
+          jwt.stubs(:header).returns('Mailroom-Api-Request')
+          jwt.stubs(:token).returns('a_jwt_token')
+
+          delivery = MailRoom::Delivery::Postback.new(delivery_options)
+
+          delivery.deliver('a message')
+          expect(request.headers['Mailroom-Api-Request']).to eql('a_jwt_token')
+
+          # A different jwt token for the second time
+          jwt.stubs(:token).returns('another_jwt_token')
+
+          delivery.deliver('another message')
+          expect(request.headers['Mailroom-Api-Request']).to eql('another_jwt_token')
+        end
+      end
     end
   end
 end