gitlab-mail_room 0.0.9 → 0.0.23

data/README.md

@@ -20,7 +20,14 @@ The fork is useful as we can post quick fixes to our own fork and release fixes
 
 ## README
 
-mail_room is a configuration based process that will idle on IMAP connections and execute a delivery method when a new message is received. Examples of delivery methods include:
+mail_room is a configuration based process that will listen for incoming
+e-mail and execute a delivery method when a new message is
+received. mail_room supports the following methods for receiving e-mail:
+
+* IMAP
+* [Microsoft Graph API](https://docs.microsoft.com/en-us/graph/api/resources/mail-api-overview?view=graph-rest-1.0)
+
+Examples of delivery methods include:
 
 * POST to a delivery URL (Postback)
 * Queue a job to Sidekiq or Que for later processing (Sidekiq or Que)
@@ -56,8 +63,8 @@ You will also need to install `faraday` or `letter_opener` if you use the `postb
 ```yaml
 ---
 :health_check:
-  - :address: "127.0.0.1"
-    :port: 8080
+  :address: "127.0.0.1"
+  :port: 8080
 :mailboxes:
   -
     :email: "user1@gmail.com"
@@ -117,6 +124,33 @@ You will also need to install `faraday` or `letter_opener` if you use the `postb
           :host: 127.0.0.1
           :port: 26379
       :worker: EmailReceiverWorker
+  -
+    :email: "user7@outlook365.com"
+    :password: "password"
+    :name: "inbox"
+    :inbox_method: microsoft_graph
+    :inbox_options:
+      :tenant_id: 12345
+      :client_id: ABCDE
+      :client_secret: YOUR-SECRET-HERE
+      :poll_interval: 60
+      :azure_ad_endpoint: https://login.microsoftonline.com
+      :graph_endpoint: https://graph.microsoft.com
+    :delivery_method: sidekiq
+    :delivery_options:
+      :redis_url: redis://localhost:6379
+      :worker: EmailReceiverWorker
+  -
+    :email: "user8@gmail.com"
+    :password: "password"
+    :name: "inbox"
+    :delivery_method: postback
+    :delivery_options:
+      :delivery_url: "http://localhost:3000/inbox"
+      :jwt_auth_header: "Mailroom-Api-Request"
+      :jwt_issuer: "mailroom"
+      :jwt_algorithm: "HS256"
+      :jwt_secret_path: "/etc/secrets/mailroom/.mailroom_secret"
 ```
 
 **Note:** If using `delete_after_delivery`, you also probably want to use
@@ -134,6 +168,91 @@ the server is running. Otherwise, it returns a 500 status code.
 
 This feature is not included in upstream `mail_room` and is specific to GitLab.
 
+## inbox_method
+
+By default, IMAP mode is assumed for reading a mailbox.
+
+### IMAP Server Configuration ##
+
+You can set per-mailbox configuration for the IMAP server's `host` (default: 'imap.gmail.com'), `port` (default: 993), `ssl` (default: true), and `start_tls` (default: false).
+
+If you want to set additional options for IMAP SSL you can pass a YAML hash to match [SSLContext#set_params](http://docs.ruby-lang.org/en/2.2.0/OpenSSL/SSL/SSLContext.html#method-i-set_params). If you set `verify_mode` to `:none` it'll replace with the appropriate constant.
+
+If you're seeing the error `Please log in via your web browser: https://support.google.com/mail/accounts/answer/78754 (Failure)`, you need to configure your Gmail account to allow less secure apps to access it: https://support.google.com/accounts/answer/6010255.
+
+### Microsoft Graph configuration
+
+To use the Microsoft Graph API instead of IMAP to read e-mail, you will
+need to create an application in the Azure Active Directory. See the
+[Microsoft instructions](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app) for more details:
+
+1. Sign in to the [Azure portal](https://portal.azure.com).
+1. Search for and select `Azure Active Directory`.
+1. Under `Manage`, select `App registrations` > `New registration`.
+1. Enter a `Name` for your application, such as `MailRoom`. Users of your app might see this name, and you can change it later.
+1. If `Supported account types` is listed, select the appropriate option.
+1. Leave `Redirect URI` blank. This is not needed.
+1. Select `Register`.
+1. Under `Manage`, select `Certificates & secrets`.
+1. Under `Client secrets`, select `New client secret`, and enter a name.
+1. Under `Expires`, select `Never`, unless you plan on updating the credentials every time it expires.
+1. Select `Add`. Record the secret value in a safe location for use in a later step.
+1. Under `Manage`, select `API Permissions` > `Add a permission`. Select `Microsoft Graph`.
+1. Select `Application permissions`.
+1. Under the `Mail` node, select `Mail.ReadWrite`, and then select Add permissions.
+1. If `User.Read` is listed in the permission list, you can delete this.
+1. Click `Grant admin consent` for these permissions.
+
+#### Restrict mailbox access
+
+Note that for MailRoom to work as a service account, this application
+must have the `Mail.ReadWrite` to read/write mail in *all*
+mailboxes. However, while this appears to be security risk,
+we can configure an application access policy to limit the
+mailbox access for this account. [Follow these instructions](https://docs.microsoft.com/en-us/graph/auth-limit-mailbox-access)
+to setup PowerShell and configure this policy.
+
+#### MailRoom config for Microsoft Graph
+
+In the MailRoom configuration, set `inbox_method` to `microsoft_graph`.
+You will also need:
+
+* The client and tenant ID from the `Overview` section in the Azure app page
+* The client secret created earlier
+
+Fill in `inbox_options` with these values:
+
+```yaml
+    :inbox_method: microsoft_graph
+    :inbox_options:
+      :tenant_id: 12345
+      :client_id: ABCDE
+      :client_secret: YOUR-SECRET-HERE
+      :poll_interval: 60
+```
+
+By default, MailRoom will poll for new messages every 60 seconds. `poll_interval` configures the number of
+seconds to poll. Setting the value to 0 or under will default to 60 seconds.
+
+### Alternative Azure cloud deployments
+
+MailRoom will default to using the standard Azure HTTPS endpoints. To
+configure MailRoom with Microsoft Cloud for US Government or other
+[national cloud deployments](https://docs.microsoft.com/en-us/graph/deployments), set
+the `azure_ad_endpoint` and `graph_endpoint` accordingly. For example,
+for Microsoft Cloud for US Government:
+
+```yaml
+    :inbox_method: microsoft_graph
+    :inbox_options:
+      :tenant_id: 12345
+      :client_id: ABCDE
+      :client_secret: YOUR-SECRET-HERE
+      :poll_interval: 60
+      :azure_ad_endpoint: https://login.microsoftonline.us
+      :graph_endpoint: https://graph.microsoft.us
+```
+
 ## delivery_method ##
 
 ### postback ###
@@ -260,8 +379,8 @@ And finally, configure MailRoom to use the postback configuration with the optio
 :delivery_method: postback
 :delivery_options:
   :delivery_url: https://example.com/rails/action_mailbox/relay/inbound_emails
-  :delivery_username: actionmailbox
-  :delivery_password: <INGRESS_PASSWORD>
+  :username: actionmailbox
+  :password: <INGRESS_PASSWORD>
 ```
 
 ## Receiving `postback` in Rails ##
@@ -278,20 +397,12 @@ it's probably because the content-type is set to Faraday's default, which is  `H
 ## idle_timeout ##
 
 By default, the IDLE command will wait for 29 minutes (in order to keep the server connection happy).
-If you'd prefer not to wait that long, you can pass `imap_timeout` in seconds for your mailbox configuration.
+If you'd prefer not to wait that long, you can pass `idle_timeout` in seconds for your mailbox configuration.
 
 ## Search Command ##
 
 This setting allows configuration of the IMAP search command sent to the server. This still defaults 'UNSEEN'. You may find that 'NEW' works better for you.
 
-## IMAP Server Configuration ##
-
-You can set per-mailbox configuration for the IMAP server's `host` (default: 'imap.gmail.com'), `port` (default: 993), `ssl` (default: true), and `start_tls` (default: false).
-
-If you want to set additional options for IMAP SSL you can pass a YAML hash to match [SSLContext#set_params](http://docs.ruby-lang.org/en/2.2.0/OpenSSL/SSL/SSLContext.html#method-i-set_params). If you set `verify_mode` to `:none` it'll replace with the appropriate constant.
-
-If you're seeing the error `Please log in via your web browser: https://support.google.com/mail/accounts/answer/78754 (Failure)`, you need to configure your Gmail account to allow less secure apps to access it: https://support.google.com/accounts/answer/6010255.
-
 ## Running in Production ##
 
 I suggest running with either upstart or init.d. Check out this wiki page for some example scripts for both: https://github.com/tpitale/mail_room/wiki/Init-Scripts-for-Running-mail_room

data/Rakefile

@@ -3,4 +3,4 @@ require 'rspec/core/rake_task'
 
 RSpec::Core::RakeTask.new(:spec)
 
-task :default => :spec
+task default: :spec

data/lib/mail_room/arbitration/redis.rb

@@ -31,7 +31,7 @@ module MailRoom
         # Any subsequent failure in the instance which gets the lock will be dealt
         # with by the expiration, at which time another instance can pick up the
         # message and try again.
-        client.set(key, 1, {:nx => true, :ex => expiration})
+        client.set(key, 1, nx: true, ex: expiration)
       end
 
       private

data/lib/mail_room/connection.rb

@@ -6,18 +6,23 @@ module MailRoom
 
     def initialize(mailbox)
       @mailbox = mailbox
+      @stopped = false
     end
 
     def on_new_message(&block)
       @new_message_handler = block
     end
 
+    def stopped?
+      @stopped
+    end
+
     def wait
       raise NotImplementedError
     end
 
     def quit
-      raise NotImplementedError
+      @stopped = true
     end
   end
 end

data/lib/mail_room/crash_handler.rb

@@ -1,3 +1,4 @@
+require 'date'
 
 module MailRoom
   class CrashHandler
@@ -19,7 +20,7 @@ module MailRoom
     private
 
     def json(error)
-      { time: Time.now, severity: :fatal, message: error.message, backtrace: error.backtrace }.to_json
+      { time: DateTime.now.iso8601(3), severity: :fatal, message: error.message, backtrace: error.backtrace }.to_json
     end
   end
 end

data/lib/mail_room/delivery/letter_opener.rb

@@ -24,7 +24,7 @@ module MailRoom
       # Trigger `LetterOpener` to deliver our message
       # @param message [String] the email message as a string, RFC822 format
       def deliver(message)
-        method = ::LetterOpener::DeliveryMethod.new(:location => @delivery_options.location)
+        method = ::LetterOpener::DeliveryMethod.new(location: @delivery_options.location)
         method.deliver!(Mail.read_from_string(message))
 
         true

data/lib/mail_room/delivery/postback.rb

@@ -1,11 +1,12 @@
 require 'faraday'
+require "mail_room/jwt"
 
 module MailRoom
   module Delivery
     # Postback Delivery method
     # @author Tony Pitale
     class Postback
-      Options = Struct.new(:url, :token, :username, :password, :logger, :content_type) do
+      Options = Struct.new(:url, :token, :username, :password, :logger, :content_type, :jwt) do
         def initialize(mailbox)
           url =
             mailbox.delivery_url ||
@@ -17,23 +18,44 @@ module MailRoom
             mailbox.delivery_options[:delivery_token] ||
             mailbox.delivery_options[:token]
 
-          username = mailbox.delivery_options[:username]
-          password = mailbox.delivery_options[:password]
+          jwt = initialize_jwt(mailbox.delivery_options)
+
+          username = 
+            mailbox.delivery_options[:username] ||
+            mailbox.delivery_options[:delivery_username]
+          password = 
+            mailbox.delivery_options[:password] ||
+            mailbox.delivery_options[:delivery_password]
 
           logger = mailbox.logger
 
           content_type = mailbox.delivery_options[:content_type]
 
-          super(url, token, username, password, logger, content_type)
+          super(url, token, username, password, logger, content_type, jwt)
         end
 
         def token_auth?
           !self[:token].nil?
         end
 
+        def jwt_auth?
+          self[:jwt].valid?
+        end
+
         def basic_auth?
           !self[:username].nil? && !self[:password].nil?
         end
+
+        private
+
+        def initialize_jwt(delivery_options)
+          ::MailRoom::JWT.new(
+            header: delivery_options[:jwt_auth_header],
+            secret_path: delivery_options[:jwt_secret_path],
+            algorithm: delivery_options[:jwt_algorithm],
+            issuer: delivery_options[:jwt_issuer]
+          )
+        end
       end
 
       # Build a new delivery, hold the delivery options
@@ -60,13 +82,27 @@ module MailRoom
         connection.post do |request|
           request.url @delivery_options.url
           request.body = message
-          # request.options[:timeout] = 3
-          request.headers['Content-Type'] = @delivery_options.content_type unless @delivery_options.content_type.nil?
+          config_request_content_type(request)
+          config_request_jwt_auth(request)
         end
 
         @delivery_options.logger.info({ delivery_method: 'Postback', action: 'message pushed', url: @delivery_options.url })
         true
       end
+
+      private
+
+      def config_request_content_type(request)
+        return if @delivery_options.content_type.nil?
+
+        request.headers['Content-Type'] = @delivery_options.content_type
+      end
+
+      def config_request_jwt_auth(request)
+        return unless @delivery_options.jwt_auth?
+
+        request.headers[@delivery_options.jwt.header] = @delivery_options.jwt.token
+      end
     end
   end
 end

data/lib/mail_room/delivery/que.rb

@@ -1,5 +1,6 @@
 require 'pg'
 require 'json'
+require 'charlock_holmes'
 
 module MailRoom
   module Delivery
@@ -34,7 +35,7 @@ module MailRoom
       # deliver the message by pushing it onto the configured Sidekiq queue
       # @param message [String] the email message as a string, RFC822 format
       def deliver(message)
-        queue_job(message)
+        queue_job(utf8_encode_message(message))
         @options.logger.info({ delivery_method: 'Que', action: 'message pushed' })
       end
 
@@ -58,6 +59,19 @@ module MailRoom
 
         connection.exec(sql, [options.priority, options.job_class, options.queue, JSON.dump(args)])
       end
+
+      def utf8_encode_message(message)
+        message = message.dup
+
+        message.force_encoding("UTF-8")
+        return message if message.valid_encoding?
+
+        detection = CharlockHolmes::EncodingDetector.detect(message)
+        return message unless detection && detection[:encoding]
+
+        # Convert non-UTF-8 body UTF-8 so it can be dumped as JSON.
+        CharlockHolmes::Converter.convert(message, detection[:encoding], 'UTF-8')
+      end
     end
   end
 end

data/lib/mail_room/delivery/sidekiq.rb

@@ -8,16 +8,17 @@ module MailRoom
     # Sidekiq Delivery method
     # @author Douwe Maan
     class Sidekiq
-      Options = Struct.new(:redis_url, :namespace, :sentinels, :queue, :worker, :logger) do
+      Options = Struct.new(:redis_url, :namespace, :sentinels, :queue, :worker, :logger, :redis_db) do
         def initialize(mailbox)
           redis_url = mailbox.delivery_options[:redis_url] || "redis://localhost:6379"
+          redis_db  = mailbox.delivery_options[:redis_db] || 0
           namespace = mailbox.delivery_options[:namespace]
           sentinels = mailbox.delivery_options[:sentinels]
           queue     = mailbox.delivery_options[:queue] || "default"
           worker    = mailbox.delivery_options[:worker]
           logger = mailbox.logger
 
-          super(redis_url, namespace, sentinels, queue, worker, logger)
+          super(redis_url, namespace, sentinels, queue, worker, logger, redis_db)
         end
       end
 
@@ -45,7 +46,7 @@ module MailRoom
       def client
         @client ||= begin
           sentinels = options.sentinels
-          redis_options = { url: options.redis_url }
+          redis_options = { url: options.redis_url, db: options.redis_db }
           redis_options[:sentinels] = sentinels if sentinels
 
           redis = ::Redis.new(redis_options)

data/lib/mail_room/jwt.rb

@@ -0,0 +1,39 @@
+# frozen_string_literal: true
+
+require 'faraday'
+require 'securerandom'
+require 'jwt'
+require 'base64'
+
+module MailRoom
+  # Responsible for validating and generating JWT token
+  class JWT
+    DEFAULT_ISSUER = 'mailroom'
+    DEFAULT_ALGORITHM = 'HS256'
+
+    attr_reader :header, :secret_path, :issuer, :algorithm
+
+    def initialize(header:, secret_path:, issuer:, algorithm:)
+      @header = header
+      @secret_path = secret_path
+      @issuer = issuer || DEFAULT_ISSUER
+      @algorithm = algorithm || DEFAULT_ALGORITHM
+    end
+
+    def valid?
+      [@header, @secret_path, @issuer, @algorithm].none?(&:nil?)
+    end
+
+    def token
+      return nil unless valid?
+
+      secret = Base64.strict_decode64(File.read(@secret_path).chomp)
+      payload = {
+        nonce: SecureRandom.hex(12),
+        iat: Time.now.to_i, # https://github.com/jwt/ruby-jwt#issued-at-claim
+        iss: @issuer
+      }
+      ::JWT.encode payload, secret, @algorithm
+    end
+  end
+end

data/lib/mail_room/logger/structured.rb

@@ -1,3 +1,4 @@
+require 'date'
 require 'logger'
 require 'json'
 
@@ -10,12 +11,25 @@ module MailRoom
 
         data = {}
         data[:severity] = severity
-        data[:time] = timestamp || Time.now.to_s
+        data[:time] = format_timestamp(timestamp || Time.now)
         # only accept a Hash
         data.merge!(message)
 
         data.to_json + "\n"
       end
+
+      private
+
+      def format_timestamp(timestamp)
+        case timestamp
+        when Time
+          timestamp.to_datetime.iso8601(3).to_s
+        when DateTime
+          timestamp.iso8601(3).to_s
+        else
+          timestamp
+        end
+      end
     end
   end
 end

data/lib/mail_room/mailbox.rb

@@ -1,11 +1,14 @@
 require "mail_room/delivery"
 require "mail_room/arbitration"
 require "mail_room/imap"
+require "mail_room/microsoft_graph"
 
 module MailRoom
   # Mailbox Configuration fields
   MAILBOX_FIELDS = [
     :email,
+    :inbox_method,
+    :inbox_options,
     :password,
     :host,
     :port,
@@ -42,24 +45,26 @@ module MailRoom
     # 29 minutes, as suggested by the spec: https://tools.ietf.org/html/rfc2177
     IMAP_IDLE_TIMEOUT = 29 * 60 # 29 minutes in in seconds
 
-    REQUIRED_CONFIGURATION = [:name, :email, :password, :host, :port]
+    IMAP_CONFIGURATION = [:name, :email, :password, :host, :port].freeze
+    MICROSOFT_GRAPH_CONFIGURATION = [:name, :email].freeze
+    MICROSOFT_GRAPH_INBOX_OPTIONS = [:tenant_id, :client_id, :client_secret].freeze
 
     # Default attributes for the mailbox configuration
     DEFAULTS = {
-      :search_command => 'UNSEEN',
-      :delivery_method => 'postback',
-      :host => 'imap.gmail.com',
-      :port => 993,
-      :ssl => true,
-      :start_tls => false,
-      :limit_max_unread => 0,
-      :idle_timeout => IMAP_IDLE_TIMEOUT,
-      :delete_after_delivery => false,
-      :expunge_deleted => false,
-      :delivery_options => {},
-      :arbitration_method => 'noop',
-      :arbitration_options => {},
-      :logger => {}
+      search_command: 'UNSEEN',
+      delivery_method: 'postback',
+      host: 'imap.gmail.com',
+      port: 993,
+      ssl: true,
+      start_tls: false,
+      limit_max_unread: 0,
+      idle_timeout: IMAP_IDLE_TIMEOUT,
+      delete_after_delivery: false,
+      expunge_deleted: false,
+      delivery_options: {},
+      arbitration_method: 'noop',
+      arbitration_options: {},
+      logger: {}
     }
 
     # Store the configuration and require the appropriate delivery method
@@ -122,14 +127,32 @@ module MailRoom
       { email: self.email, name: self.name }
     end
 
+    def imap?
+      !microsoft_graph?
+    end
+
+    def microsoft_graph?
+      self[:inbox_method].to_s == 'microsoft_graph'
+    end
+
     def validate!
+      if microsoft_graph?
+        validate_microsoft_graph!
+      else
+        validate_imap!
+      end
+    end
+
+    private
+
+    def validate_imap!
       if self[:idle_timeout] > IMAP_IDLE_TIMEOUT
         raise IdleTimeoutTooLarge,
               "Please use an idle timeout smaller than #{29*60} to prevent " \
               "IMAP server disconnects"
       end
 
-      REQUIRED_CONFIGURATION.each do |k|
+      IMAP_CONFIGURATION.each do |k|
         if self[k].nil?
           raise ConfigurationError,
                 "Field :#{k} is required in Mailbox: #{inspect}"
@@ -137,7 +160,23 @@ module MailRoom
       end
     end
 
-    private
+    def validate_microsoft_graph!
+      raise ConfigurationError, "Missing inbox_options in Mailbox: #{inspect}" unless self.inbox_options.is_a?(Hash)
+
+      MICROSOFT_GRAPH_CONFIGURATION.each do |k|
+        if self[k].nil?
+          raise ConfigurationError,
+                "Field :#{k} is required in Mailbox: #{inspect}"
+        end
+      end
+
+      MICROSOFT_GRAPH_INBOX_OPTIONS.each do |k|
+        if self[:inbox_options][k].nil?
+          raise ConfigurationError,
+                "inbox_options field :#{k} is required in Mailbox: #{inspect}"
+        end
+      end
+    end
 
     def parsed_arbitration_options
       arbitration_klass::Options.new(self)

data/lib/mail_room/mailbox_watcher.rb

@@ -62,8 +62,14 @@ module MailRoom
     end
 
     private
+
     def connection
-      @connection ||= ::MailRoom::IMAP::Connection.new(@mailbox)
+      @connection ||=
+        if @mailbox.microsoft_graph?
+          ::MailRoom::MicrosoftGraph::Connection.new(@mailbox)
+        else
+          ::MailRoom::IMAP::Connection.new(@mailbox)
+        end
     end
   end
 end