gitlab-mail_room 0.0.4 → 0.0.20

data/README.md

@@ -20,7 +20,14 @@ The fork is useful as we can post quick fixes to our own fork and release fixes
 
 ## README
 
-mail_room is a configuration based process that will idle on IMAP connections and execute a delivery method when a new message is received. Examples of delivery methods include:
+mail_room is a configuration based process that will listen for incoming
+e-mail and execute a delivery method when a new message is
+received. mail_room supports the following methods for receiving e-mail:
+
+* IMAP
+* [Microsoft Graph API](https://docs.microsoft.com/en-us/graph/api/resources/mail-api-overview?view=graph-rest-1.0)
+
+Examples of delivery methods include:
 
 * POST to a delivery URL (Postback)
 * Queue a job to Sidekiq or Que for later processing (Sidekiq or Que)
@@ -55,6 +62,9 @@ You will also need to install `faraday` or `letter_opener` if you use the `postb
 
 ```yaml
 ---
+:health_check:
+  :address: "127.0.0.1"
+  :port: 8080
 :mailboxes:
   -
     :email: "user1@gmail.com"
@@ -114,11 +124,135 @@ You will also need to install `faraday` or `letter_opener` if you use the `postb
           :host: 127.0.0.1
           :port: 26379
       :worker: EmailReceiverWorker
+  -
+    :email: "user7@outlook365.com"
+    :password: "password"
+    :name: "inbox"
+    :inbox_method: microsoft_graph
+    :inbox_options:
+      :tenant_id: 12345
+      :client_id: ABCDE
+      :client_secret: YOUR-SECRET-HERE
+      :poll_interval: 60
+      :azure_ad_endpoint: https://login.microsoftonline.com
+      :graph_endpoint: https://graph.microsoft.com
+    :delivery_method: sidekiq
+    :delivery_options:
+      :redis_url: redis://localhost:6379
+      :worker: EmailReceiverWorker
+  -
+    :email: "user8@gmail.com"
+    :password: "password"
+    :name: "inbox"
+    :delivery_method: postback
+    :delivery_options:
+      :delivery_url: "http://localhost:3000/inbox"
+      :jwt_auth_header: "Mailroom-Api-Request"
+      :jwt_issuer: "mailroom"
+      :jwt_algorithm: "HS256"
+      :jwt_secret_path: "/etc/secrets/mailroom/.mailroom_secret"
 ```
 
 **Note:** If using `delete_after_delivery`, you also probably want to use
 `expunge_deleted` unless you really know what you're doing.
 
+## health_check ##
+
+Requires `webrick` gem to be installed.
+
+This option enables an HTTP server that listens to a bind address
+defined by `address` and `port`. The following endpoints are supported:
+
+* `/liveness`: This returns a 200 status code with `OK` as the body if
+the server is running. Otherwise, it returns a 500 status code.
+
+This feature is not included in upstream `mail_room` and is specific to GitLab.
+
+## inbox_method
+
+By default, IMAP mode is assumed for reading a mailbox.
+
+### IMAP Server Configuration ##
+
+You can set per-mailbox configuration for the IMAP server's `host` (default: 'imap.gmail.com'), `port` (default: 993), `ssl` (default: true), and `start_tls` (default: false).
+
+If you want to set additional options for IMAP SSL you can pass a YAML hash to match [SSLContext#set_params](http://docs.ruby-lang.org/en/2.2.0/OpenSSL/SSL/SSLContext.html#method-i-set_params). If you set `verify_mode` to `:none` it'll replace with the appropriate constant.
+
+If you're seeing the error `Please log in via your web browser: https://support.google.com/mail/accounts/answer/78754 (Failure)`, you need to configure your Gmail account to allow less secure apps to access it: https://support.google.com/accounts/answer/6010255.
+
+### Microsoft Graph configuration
+
+To use the Microsoft Graph API instead of IMAP to read e-mail, you will
+need to create an application in the Azure Active Directory. See the
+[Microsoft instructions](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app) for more details:
+
+1. Sign in to the [Azure portal](https://portal.azure.com).
+1. Search for and select `Azure Active Directory`.
+1. Under `Manage`, select `App registrations` > `New registration`.
+1. Enter a `Name` for your application, such as `MailRoom`. Users of your app might see this name, and you can change it later.
+1. If `Supported account types` is listed, select the appropriate option.
+1. Leave `Redirect URI` blank. This is not needed.
+1. Select `Register`.
+1. Under `Manage`, select `Certificates & secrets`.
+1. Under `Client secrets`, select `New client secret`, and enter a name.
+1. Under `Expires`, select `Never`, unless you plan on updating the credentials every time it expires.
+1. Select `Add`. Record the secret value in a safe location for use in a later step.
+1. Under `Manage`, select `API Permissions` > `Add a permission`. Select `Microsoft Graph`.
+1. Select `Application permissions`.
+1. Under the `Mail` node, select `Mail.ReadWrite`, and then select Add permissions.
+1. If `User.Read` is listed in the permission list, you can delete this.
+1. Click `Grant admin consent` for these permissions.
+
+#### Restrict mailbox access
+
+Note that for MailRoom to work as a service account, this application
+must have the `Mail.ReadWrite` to read/write mail in *all*
+mailboxes. However, while this appears to be security risk,
+we can configure an application access policy to limit the
+mailbox access for this account. [Follow these instructions](https://docs.microsoft.com/en-us/graph/auth-limit-mailbox-access)
+to setup PowerShell and configure this policy.
+
+#### MailRoom config for Microsoft Graph
+
+In the MailRoom configuration, set `inbox_method` to `microsoft_graph`.
+You will also need:
+
+* The client and tenant ID from the `Overview` section in the Azure app page
+* The client secret created earlier
+
+Fill in `inbox_options` with these values:
+
+```yaml
+    :inbox_method: microsoft_graph
+    :inbox_options:
+      :tenant_id: 12345
+      :client_id: ABCDE
+      :client_secret: YOUR-SECRET-HERE
+      :poll_interval: 60
+```
+
+By default, MailRoom will poll for new messages every 60 seconds. `poll_interval` configures the number of
+seconds to poll. Setting the value to 0 or under will default to 60 seconds.
+
+### Alternative Azure cloud deployments
+
+MailRoom will default to using the standard Azure HTTPS endpoints. To
+configure MailRoom with Microsoft Cloud for US Government or other
+[national cloud deployments](https://docs.microsoft.com/en-us/graph/deployments), set
+the `azure_ad_endpoint` and `graph_endpoint` accordingly. For example,
+for Microsoft Cloud for US Government:
+
+```yaml
+    :inbox_method: microsoft_graph
+    :inbox_options:
+      :tenant_id: 12345
+      :client_id: ABCDE
+      :client_secret: YOUR-SECRET-HERE
+      :poll_interval: 60
+      :azure_ad_endpoint: https://login.microsoftonline.us
+      :graph_endpoint: https://graph.microsoft.us
+```
+
 ## delivery_method ##
 
 ### postback ###
@@ -263,20 +397,12 @@ it's probably because the content-type is set to Faraday's default, which is  `H
 ## idle_timeout ##
 
 By default, the IDLE command will wait for 29 minutes (in order to keep the server connection happy).
-If you'd prefer not to wait that long, you can pass `imap_timeout` in seconds for your mailbox configuration.
+If you'd prefer not to wait that long, you can pass `idle_timeout` in seconds for your mailbox configuration.
 
 ## Search Command ##
 
 This setting allows configuration of the IMAP search command sent to the server. This still defaults 'UNSEEN'. You may find that 'NEW' works better for you.
 
-## IMAP Server Configuration ##
-
-You can set per-mailbox configuration for the IMAP server's `host` (default: 'imap.gmail.com'), `port` (default: 993), `ssl` (default: true), and `start_tls` (default: false).
-
-If you want to set additional options for IMAP SSL you can pass a YAML hash to match [SSLContext#set_params](http://docs.ruby-lang.org/en/2.2.0/OpenSSL/SSL/SSLContext.html#method-i-set_params). If you set `verify_mode` to `:none` it'll replace with the appropriate constant.
-
-If you're seeing the error `Please log in via your web browser: https://support.google.com/mail/accounts/answer/78754 (Failure)`, you need to configure your Gmail account to allow less secure apps to access it: https://support.google.com/accounts/answer/6010255.
-
 ## Running in Production ##
 
 I suggest running with either upstart or init.d. Check out this wiki page for some example scripts for both: https://github.com/tpitale/mail_room/wiki/Init-Scripts-for-Running-mail_room

data/Rakefile

@@ -3,4 +3,4 @@ require 'rspec/core/rake_task'
 
 RSpec::Core::RakeTask.new(:spec)
 
-task :default => :spec
+task default: :spec

data/lib/mail_room/arbitration/redis.rb

@@ -31,7 +31,7 @@ module MailRoom
         # Any subsequent failure in the instance which gets the lock will be dealt
         # with by the expiration, at which time another instance can pick up the
         # message and try again.
-        client.set(key, 1, {:nx => true, :ex => expiration})
+        client.set(key, 1, nx: true, ex: expiration)
       end
 
       private

data/lib/mail_room/cli.rb

@@ -42,7 +42,7 @@ module MailRoom
       end.parse!(args)
 
       self.configuration = Configuration.new(options)
-      self.coordinator = Coordinator.new(configuration.mailboxes)
+      self.coordinator = Coordinator.new(configuration.mailboxes, configuration.health_check)
     end
 
     # Start the coordinator running, sets up signal traps
@@ -57,7 +57,7 @@ module MailRoom
 
       coordinator.run
     rescue Exception => e # not just Errors, but includes lower-level Exceptions
-      CrashHandler.new(error: e, format: @options[:exit_error_format]).handle
+      CrashHandler.new.handle(e, @options[:exit_error_format])
       exit
     end
   end

data/lib/mail_room/configuration.rb

@@ -4,7 +4,7 @@ module MailRoom
   # Wraps configuration for a set of individual mailboxes with global config
   # @author Tony Pitale
   class Configuration
-    attr_accessor :mailboxes, :log_path, :quiet
+    attr_accessor :mailboxes, :log_path, :quiet, :health_check
 
     # Initialize a new configuration of mailboxes
     def initialize(options={})
@@ -18,6 +18,7 @@ module MailRoom
           config_file = YAML.load(erb.result)
 
           set_mailboxes(config_file[:mailboxes])
+          set_health_check(config_file[:health_check])
         rescue => e
           raise e unless quiet
         end
@@ -32,5 +33,14 @@ module MailRoom
         self.mailboxes << Mailbox.new(attributes)
       end
     end
+
+    # Builds the health checker from YAML configuration
+    #
+    # @param health_check_config nil or a Hash containing :address and :port
+    def set_health_check(health_check_config)
+      return unless health_check_config
+
+      self.health_check = HealthCheck.new(health_check_config)
+    end
   end
 end

data/lib/mail_room/connection.rb

@@ -1,195 +1,28 @@
+# frozen_string_literal: true
+
 module MailRoom
   class Connection
+    attr_reader :mailbox, :new_message_handler
+
     def initialize(mailbox)
       @mailbox = mailbox
-
-      # log in and set the mailbox
-      reset
-      setup
+      @stopped = false
     end
 
     def on_new_message(&block)
       @new_message_handler = block
     end
 
-    # is the connection logged in?
-    # @return [Boolean]
-    def logged_in?
-      @logged_in
-    end
-
-    # is the connection blocked idling?
-    # @return [Boolean]
-    def idling?
-      @idling
-    end
-
-    # is the imap connection closed?
-    # @return [Boolean]
-    def disconnected?
-      @imap.disconnected?
-    end
-
-    # is the connection ready to idle?
-    # @return [Boolean]
-    def ready_to_idle?
-      logged_in? && !idling?
-    end
-
-    def quit
-      stop_idling
-      reset
+    def stopped?
+      @stopped
     end
 
     def wait
-      begin
-        # in case we missed any between idles
-        process_mailbox
-
-        idle
-
-        process_mailbox
-      rescue Net::IMAP::Error, IOError
-        @mailbox.logger.warn({ context: @mailbox.context, action: "Disconnected. Resetting..." })
-        reset
-        setup
-      end
-    end
-
-    private
-
-    def reset
-      @imap = nil
-      @logged_in = false
-      @idling = false
-    end
-
-    def setup
-      @mailbox.logger.info({ context: @mailbox.context, action: "Starting TLS session" })
-      start_tls
-
-      @mailbox.logger.info({ context: @mailbox.context, action: "Logging into mailbox" })
-      log_in
-
-      @mailbox.logger.info({ context: @mailbox.context, action: "Setting mailbox" })
-      set_mailbox
-    end
-
-    # build a net/imap connection to google imap
-    def imap
-      @imap ||= Net::IMAP.new(@mailbox.host, :port => @mailbox.port, :ssl => @mailbox.ssl_options)
-    end
-
-    # start a TLS session
-    def start_tls
-      imap.starttls if @mailbox.start_tls
-    end
-
-    # send the imap login command to google
-    def log_in
-      imap.login(@mailbox.email, @mailbox.password)
-      @logged_in = true
-    end
-
-    # select the mailbox name we want to use
-    def set_mailbox
-      imap.select(@mailbox.name) if logged_in?
-    end
-
-    # is the response for a new message?
-    # @param response [Net::IMAP::TaggedResponse] the imap response from idle
-    # @return [Boolean]
-    def message_exists?(response)
-      response.respond_to?(:name) && response.name == 'EXISTS'
-    end
-
-    # @private
-    def idle_handler
-      lambda {|response| imap.idle_done if message_exists?(response)}
-    end
-
-    # maintain an imap idle connection
-    def idle
-      return unless ready_to_idle?
-
-      @mailbox.logger.info({ context: @mailbox.context, action: "Idling" })
-      @idling = true
-
-      imap.idle(@mailbox.idle_timeout, &idle_handler)
-    ensure
-      @idling = false
-    end
-
-    # trigger the idle to finish and wait for the thread to finish
-    def stop_idling
-      return unless idling?
-
-      imap.idle_done
-
-      # idling_thread.join
-      # self.idling_thread = nil
-    end
-
-    def process_mailbox
-      return unless @new_message_handler
-      @mailbox.logger.info({ context: @mailbox.context, action: "Processing started" })
-
-      msgs = new_messages
-
-      any_deletions = msgs.
-                      # deliver each new message, collect success
-                      map(&@new_message_handler).
-                      # include messages with success
-                      zip(msgs).
-                      # filter failed deliveries, collect message
-                      select(&:first).map(&:last).
-                      # scrub delivered messages
-                      map { |message| scrub(message) }.
-                      any?
-
-      imap.expunge if @mailbox.expunge_deleted && any_deletions
-    end
-
-    def scrub(message)
-      if @mailbox.delete_after_delivery
-        imap.store(message.seqno, "+FLAGS", [Net::IMAP::DELETED])
-        true
-      end
-    end
-
-    # @private
-    # fetch all messages for the new message ids
-    def new_messages
-      # Both of these calls may results in
-      #   imap raising an EOFError, we handle
-      #   this exception in the watcher
-      messages_for_ids(new_message_ids)
+      raise NotImplementedError
     end
 
-    # TODO: label messages?
-    #   @imap.store(id, "+X-GM-LABELS", [label])
-
-    # @private
-    # search for all new (unseen) message ids
-    # @return [Array<Integer>] message ids
-    def new_message_ids
-      # uid_search still leaves messages UNSEEN
-      all_unread = @imap.uid_search(@mailbox.search_command)
-
-      to_deliver = all_unread.select { |uid| @mailbox.deliver?(uid) }
-      @mailbox.logger.info({ context: @mailbox.context, action: "Getting new messages", unread: {count: all_unread.count, ids: all_unread}, to_be_delivered: { count: to_deliver.count, ids: all_unread } })
-      to_deliver
-    end
-
-    # @private
-    # fetch the email for all given ids in RFC822 format
-    # @param ids [Array<Integer>] list of message ids
-    # @return [Array<Net::IMAP::FetchData>] the net/imap messages for the given ids
-    def messages_for_ids(uids)
-      return [] if uids.empty?
-
-      # uid_fetch marks as SEEN, will not be re-fetched for UNSEEN
-      imap.uid_fetch(uids, "RFC822")
+    def quit
+      @stopped = true
     end
   end
 end

data/lib/mail_room/coordinator.rb

@@ -2,13 +2,15 @@ module MailRoom
   # Coordinate the mailbox watchers
   # @author Tony Pitale
   class Coordinator
-    attr_accessor :watchers, :running
+    attr_accessor :watchers, :running, :health_check
 
     # build watchers for a set of mailboxes
     # @params mailboxes [Array<MailRoom::Mailbox>] mailboxes to be watched
-    def initialize(mailboxes)
+    # @params health_check <MailRoom::HealthCheck> health checker to run
+    def initialize(mailboxes, health_check = nil)
       self.watchers = []
 
+      @health_check = health_check
       mailboxes.each {|box| self.watchers << MailboxWatcher.new(box)}
     end
 
@@ -16,10 +18,11 @@ module MailRoom
 
     # start each of the watchers to running
     def run
+      health_check&.run
       watchers.each(&:run)
-      
+
       self.running = true
-      
+
       sleep_while_running
     ensure
       quit
@@ -27,6 +30,7 @@ module MailRoom
 
     # quit each of the watchers when we're done running
     def quit
+      health_check&.quit
       watchers.each(&:quit)
     end
 

data/lib/mail_room/crash_handler.rb

@@ -1,30 +1,26 @@
+require 'date'
 
 module MailRoom
   class CrashHandler
+    SUPPORTED_FORMATS = %w[json none]
 
-    attr_reader :error, :format
-
-    SUPPORTED_FORMATS = %w[json plain]
-
-    def initialize(error:, format:)
-      @error = error
-      @format = format
+    def initialize(stream=STDOUT)
+      @stream = stream
     end
 
-    def handle
+    def handle(error, format)
       if format == 'json'
-        puts json
+        @stream.puts json(error)
         return
       end
 
-      # 'plain' is equivalent to outputting the error into stdout as-is
       raise error
     end
 
     private
 
-    def json
-      { time: Time.now, severity: :fatal, message: error.message, backtrace: error.backtrace }.to_json
+    def json(error)
+      { time: DateTime.now.iso8601(3), severity: :fatal, message: error.message, backtrace: error.backtrace }.to_json
     end
   end
 end

data/lib/mail_room/delivery/letter_opener.rb

@@ -24,7 +24,7 @@ module MailRoom
       # Trigger `LetterOpener` to deliver our message
       # @param message [String] the email message as a string, RFC822 format
       def deliver(message)
-        method = ::LetterOpener::DeliveryMethod.new(:location => @delivery_options.location)
+        method = ::LetterOpener::DeliveryMethod.new(location: @delivery_options.location)
         method.deliver!(Mail.read_from_string(message))
 
         true

data/lib/mail_room/delivery/postback.rb

@@ -1,11 +1,12 @@
 require 'faraday'
+require "mail_room/jwt"
 
 module MailRoom
   module Delivery
     # Postback Delivery method
     # @author Tony Pitale
     class Postback
-      Options = Struct.new(:url, :token, :username, :password, :logger, :content_type) do
+      Options = Struct.new(:url, :token, :username, :password, :logger, :content_type, :jwt) do
         def initialize(mailbox)
           url =
             mailbox.delivery_url ||
@@ -17,6 +18,8 @@ module MailRoom
             mailbox.delivery_options[:delivery_token] ||
             mailbox.delivery_options[:token]
 
+          jwt = initialize_jwt(mailbox.delivery_options)
+
           username = mailbox.delivery_options[:username]
           password = mailbox.delivery_options[:password]
 
@@ -24,16 +27,31 @@ module MailRoom
 
           content_type = mailbox.delivery_options[:content_type]
 
-          super(url, token, username, password, logger, content_type)
+          super(url, token, username, password, logger, content_type, jwt)
         end
 
         def token_auth?
           !self[:token].nil?
         end
 
+        def jwt_auth?
+          self[:jwt].valid?
+        end
+
         def basic_auth?
           !self[:username].nil? && !self[:password].nil?
         end
+
+        private
+
+        def initialize_jwt(delivery_options)
+          ::MailRoom::JWT.new(
+            header: delivery_options[:jwt_auth_header],
+            secret_path: delivery_options[:jwt_secret_path],
+            algorithm: delivery_options[:jwt_algorithm],
+            issuer: delivery_options[:jwt_issuer]
+          )
+        end
       end
 
       # Build a new delivery, hold the delivery options
@@ -60,13 +78,27 @@ module MailRoom
         connection.post do |request|
           request.url @delivery_options.url
           request.body = message
-          # request.options[:timeout] = 3
-          request.headers['Content-Type'] = @delivery_options.content_type unless @delivery_options.content_type.nil?
+          config_request_content_type(request)
+          config_request_jwt_auth(request)
         end
 
         @delivery_options.logger.info({ delivery_method: 'Postback', action: 'message pushed', url: @delivery_options.url })
         true
       end
+
+      private
+
+      def config_request_content_type(request)
+        return if @delivery_options.content_type.nil?
+
+        request.headers['Content-Type'] = @delivery_options.content_type
+      end
+
+      def config_request_jwt_auth(request)
+        return unless @delivery_options.jwt_auth?
+
+        request.headers[@delivery_options.jwt.header] = @delivery_options.jwt.token
+      end
     end
   end
 end

data/lib/mail_room/delivery/sidekiq.rb

@@ -8,16 +8,17 @@ module MailRoom
     # Sidekiq Delivery method
     # @author Douwe Maan
     class Sidekiq
-      Options = Struct.new(:redis_url, :namespace, :sentinels, :queue, :worker, :logger) do
+      Options = Struct.new(:redis_url, :namespace, :sentinels, :queue, :worker, :logger, :redis_db) do
         def initialize(mailbox)
           redis_url = mailbox.delivery_options[:redis_url] || "redis://localhost:6379"
+          redis_db  = mailbox.delivery_options[:redis_db] || 0
           namespace = mailbox.delivery_options[:namespace]
           sentinels = mailbox.delivery_options[:sentinels]
           queue     = mailbox.delivery_options[:queue] || "default"
           worker    = mailbox.delivery_options[:worker]
           logger = mailbox.logger
 
-          super(redis_url, namespace, sentinels, queue, worker, logger)
+          super(redis_url, namespace, sentinels, queue, worker, logger, redis_db)
         end
       end
 
@@ -45,7 +46,7 @@ module MailRoom
       def client
         @client ||= begin
           sentinels = options.sentinels
-          redis_options = { url: options.redis_url }
+          redis_options = { url: options.redis_url, db: options.redis_db }
           redis_options[:sentinels] = sentinels if sentinels
 
           redis = ::Redis.new(redis_options)