mail_room 0.10.0 → 0.10.1

data/README.md

@@ -1,6 +1,13 @@
 # mail_room #
 
-mail_room is a configuration based process that will idle on IMAP connections and execute a delivery method when a new message is received. Examples of delivery methods include:
+mail_room is a configuration based process that will listen for incoming
+e-mail and execute a delivery method when a new message is
+received. mail_room supports the following methods for receiving e-mail:
+
+* IMAP
+* [Microsoft Graph API](https://docs.microsoft.com/en-us/graph/api/resources/mail-api-overview?view=graph-rest-1.0)
+
+Examples of delivery methods include:
 
 * POST to a delivery URL (Postback)
 * Queue a job to Sidekiq or Que for later processing (Sidekiq or Que)
@@ -46,6 +53,7 @@ You will also need to install `faraday` or `letter_opener` if you use the `postb
     :delivery_options:
       :delivery_url: "http://localhost:3000/inbox"
       :delivery_token: "abcdefg"
+      :content_type: "text/plain"
 
   -
     :email: "user2@gmail.com"
@@ -93,11 +101,91 @@ You will also need to install `faraday` or `letter_opener` if you use the `postb
           :host: 127.0.0.1
           :port: 26379
       :worker: EmailReceiverWorker
+  -
+    :email: "user7@outlook365.com"
+    :password: "password"
+    :name: "inbox"
+    :inbox_method: microsoft_graph
+    :inbox_options:
+      :tenant_id: 12345
+      :client_id: ABCDE
+      :client_secret: YOUR-SECRET-HERE
+      :poll_interval: 60
+    :delivery_method: sidekiq
+    :delivery_options:
+      :redis_url: redis://localhost:6379
+      :worker: EmailReceiverWorker
 ```
 
 **Note:** If using `delete_after_delivery`, you also probably want to use
 `expunge_deleted` unless you really know what you're doing.
 
+## inbox_method
+
+By default, IMAP mode is assumed for reading a mailbox.
+
+### IMAP Server Configuration ##
+
+You can set per-mailbox configuration for the IMAP server's `host` (default: 'imap.gmail.com'), `port` (default: 993), `ssl` (default: true), and `start_tls` (default: false).
+
+If you want to set additional options for IMAP SSL you can pass a YAML hash to match [SSLContext#set_params](http://docs.ruby-lang.org/en/2.2.0/OpenSSL/SSL/SSLContext.html#method-i-set_params). If you set `verify_mode` to `:none` it'll replace with the appropriate constant.
+
+If you're seeing the error `Please log in via your web browser: https://support.google.com/mail/accounts/answer/78754 (Failure)`, you need to configure your Gmail account to allow less secure apps to access it: https://support.google.com/accounts/answer/6010255.
+
+### Microsoft Graph configuration
+
+To use the Microsoft Graph API instead of IMAP to read e-mail, you will
+need to create an application in the Azure Active Directory. See the
+[Microsoft instructions](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app) for more details:
+
+1. Sign in to the [Azure portal](https://portal.azure.com).
+1. Search for and select `Azure Active Directory`.
+1. Under `Manage`, select `App registrations` > `New registration`.
+1. Enter a `Name` for your application, such as `MailRoom`. Users of your app might see this name, and you can change it later.
+1. If `Supported account types` is listed, select the appropriate option.
+1. Leave `Redirect URI` blank. This is not needed.
+1. Select `Register`.
+1. Under `Manage`, select `Certificates & secrets`.
+1. Under `Client secrets`, select `New client secret`, and enter a name.
+1. Under `Expires`, select `Never`, unless you plan on updating the credentials every time it expires.
+1. Select `Add`. Record the secret value in a safe location for use in a later step.
+1. Under `Manage`, select `API Permissions` > `Add a permission`. Select `Microsoft Graph`.
+1. Select `Application permissions`.
+1. Under the `Mail` node, select `Mail.ReadWrite`, and then select Add permissions.
+1. If `User.Read` is listed in the permission list, you can delete this.
+1. Click `Grant admin consent` for these permissions.
+
+#### Restrict mailbox access
+
+Note that for MailRoom to work as a service account, this application
+must have the `Mail.ReadWrite` to read/write mail in *all*
+mailboxes. However, while this appears to be security risk,
+we can configure an application access policy to limit the
+mailbox access for this account. [Follow these instructions](https://docs.microsoft.com/en-us/graph/auth-limit-mailbox-access)
+to setup PowerShell and configure this policy.
+
+#### MailRoom config for Microsoft Graph
+
+In the MailRoom configuration, set `inbox_method` to `microsoft_graph`.
+You will also need:
+
+* The client and tenant ID from the `Overview` section in the Azure app page
+* The client secret created earlier
+
+Fill in `inbox_options` with these values:
+
+```yaml
+    :inbox_method: microsoft_graph
+    :inbox_options:
+      :tenant_id: 12345
+      :client_id: ABCDE
+      :client_secret: YOUR-SECRET-HERE
+      :poll_interval: 60
+```
+
+By default, MailRoom will poll for new messages every 60 seconds. `poll_interval` configures the number of
+seconds to poll. Setting the value to 0 or under will default to 60 seconds.
+
 ## delivery_method ##
 
 ### postback ###
@@ -109,10 +197,10 @@ Requires `faraday` gem be installed.
 The default delivery method, requires `delivery_url` and `delivery_token` in
 configuration.
 
+You can pass `content_type:` option to overwrite `faraday's` default content-type(`application/x-www-form-urlencoded`) for post requests, we recommend passing `text/plain` as content-type.
+
 As the postback is essentially using your app as if it were an API endpoint,
-you may need to disable forgery protection as you would with a JSON API. In
-our case, the postback is plaintext, but the protection will still need to be
-disabled.
+you may need to disable forgery protection as you would with a JSON API.
 
 ### sidekiq ###
 
@@ -236,23 +324,18 @@ disabled, you can get the raw string of the email using `request.body.read`.
 I would recommend having the `mail` gem bundled and parse the email using
 `Mail.read_from_string(request.body.read)`.
 
+*Note:* If you get the exception (`Rack::QueryParser::InvalidParameterError (invalid %-encoding...`)
+it's probably because the content-type is set to Faraday's default, which is  `HEADERS['content-type'] = 'application/x-www-form-urlencoded'`. It can cause `Rack` to crash due to `InvalidParameterError` exception. When you send a post with `application/x-www-form-urlencoded`, `Rack` will attempt to parse the input and can end up raising an exception, for example if the email that you are forwarding contain `%%` in its content or headers it will cause Rack to crash with the message above.
+
 ## idle_timeout ##
 
 By default, the IDLE command will wait for 29 minutes (in order to keep the server connection happy).
-If you'd prefer not to wait that long, you can pass `imap_timeout` in seconds for your mailbox configuration.
+If you'd prefer not to wait that long, you can pass `idle_timeout` in seconds for your mailbox configuration.
 
 ## Search Command ##
 
 This setting allows configuration of the IMAP search command sent to the server. This still defaults 'UNSEEN'. You may find that 'NEW' works better for you.
 
-## IMAP Server Configuration ##
-
-You can set per-mailbox configuration for the IMAP server's `host` (default: 'imap.gmail.com'), `port` (default: 993), `ssl` (default: true), and `start_tls` (default: false).
-
-If you want to set additional options for IMAP SSL you can pass a YAML hash to match [SSLContext#set_params](http://docs.ruby-lang.org/en/2.2.0/OpenSSL/SSL/SSLContext.html#method-i-set_params). If you set `verify_mode` to `:none` it'll replace with the appropriate constant.
-
-If you're seeing the error `Please log in via your web browser: https://support.google.com/mail/accounts/answer/78754 (Failure)`, you need to configure your Gmail account to allow less secure apps to access it: https://support.google.com/accounts/answer/6010255.
-
 ## Running in Production ##
 
 I suggest running with either upstart or init.d. Check out this wiki page for some example scripts for both: https://github.com/tpitale/mail_room/wiki/Init-Scripts-for-Running-mail_room
@@ -328,6 +411,9 @@ MailRoom will output JSON-formatted logs to give some observability into its ope
 
 Simply configure a `log_path` for the `logger` on any of your mailboxes. By default, nothing will be logged.
 
+If you wish to log to `STDOUT` or `STDERR` instead of a file, you can pass `:stdout` or `:stderr`,
+respectively and MailRoom will log there.
+
 ## Contributing ##
 
 1. Fork it

data/Rakefile

@@ -3,4 +3,4 @@ require 'rspec/core/rake_task'
 
 RSpec::Core::RakeTask.new(:spec)
 
-task :default => :spec
+task default: :spec

data/lib/mail_room/arbitration/redis.rb

@@ -31,7 +31,7 @@ module MailRoom
         # Any subsequent failure in the instance which gets the lock will be dealt
         # with by the expiration, at which time another instance can pick up the
         # message and try again.
-        client.set(key, 1, {:nx => true, :ex => expiration})
+        client.set(key, 1, {nx: true, ex: expiration})
       end
 
       private

data/lib/mail_room/cli.rb

@@ -2,14 +2,14 @@ module MailRoom
   # The CLI parses ARGV into configuration to start the coordinator with.
   # @author Tony Pitale
   class CLI
-    attr_accessor :configuration, :coordinator
+    attr_accessor :configuration, :coordinator, :options
 
     # Initialize a new CLI instance to handle option parsing from arguments
     #   into configuration to start the coordinator running on all mailboxes
     #
     # @param args [Array] `ARGV` passed from `bin/mail_room`
     def initialize(args)
-      options = {}
+      @options = {}
 
       OptionParser.new do |parser|
         parser.banner = [
@@ -25,6 +25,10 @@ module MailRoom
           options[:quiet] = true
         end
 
+        parser.on('--log-exit-as') do |format|
+          options[:exit_error_format] = 'json' unless format.nil?
+        end
+
         # parser.on("-l", "--log FILE") do |path|
         #   options[:log_path] = path
         # end
@@ -50,6 +54,9 @@ module MailRoom
       end
 
       coordinator.run
+    rescue Exception => e # not just Errors, but includes lower-level Exceptions
+      CrashHandler.new.handle(e, @options[:exit_error_format])
+      exit
     end
   end
 end

data/lib/mail_room/connection.rb

@@ -1,195 +1,21 @@
+# frozen_string_literal: true
+
 module MailRoom
   class Connection
+    attr_reader :mailbox, :new_message_handler
+
     def initialize(mailbox)
       @mailbox = mailbox
-
-      # log in and set the mailbox
-      reset
-      setup
     end
 
     def on_new_message(&block)
       @new_message_handler = block
     end
 
-    # is the connection logged in?
-    # @return [Boolean]
-    def logged_in?
-      @logged_in
-    end
-
-    # is the connection blocked idling?
-    # @return [Boolean]
-    def idling?
-      @idling
-    end
-
-    # is the imap connection closed?
-    # @return [Boolean]
-    def disconnected?
-      @imap.disconnected?
-    end
-
-    # is the connection ready to idle?
-    # @return [Boolean]
-    def ready_to_idle?
-      logged_in? && !idling?
-    end
-
-    def quit
-      stop_idling
-      reset
-    end
-
     def wait
-      begin
-        # in case we missed any between idles
-        process_mailbox
-
-        idle
-
-        process_mailbox
-      rescue Net::IMAP::Error, IOError
-        @mailbox.logger.warn({ context: @mailbox.context, action: "Disconnected. Resetting..." })
-        reset
-        setup
-      end
+      raise NotImplementedError
     end
 
-    private
-
-    def reset
-      @imap = nil
-      @logged_in = false
-      @idling = false
-    end
-
-    def setup
-      @mailbox.logger.info({ context: @mailbox.context, action: "Starting TLS session" })
-      start_tls
-
-      @mailbox.logger.info({ context: @mailbox.context, action: "Logging into mailbox" })
-      log_in
-
-      @mailbox.logger.info({ context: @mailbox.context, action: "Setting mailbox" })
-      set_mailbox
-    end
-
-    # build a net/imap connection to google imap
-    def imap
-      @imap ||= Net::IMAP.new(@mailbox.host, :port => @mailbox.port, :ssl => @mailbox.ssl_options)
-    end
-
-    # start a TLS session
-    def start_tls
-      imap.starttls if @mailbox.start_tls
-    end
-
-    # send the imap login command to google
-    def log_in
-      imap.login(@mailbox.email, @mailbox.password)
-      @logged_in = true
-    end
-
-    # select the mailbox name we want to use
-    def set_mailbox
-      imap.select(@mailbox.name) if logged_in?
-    end
-
-    # is the response for a new message?
-    # @param response [Net::IMAP::TaggedResponse] the imap response from idle
-    # @return [Boolean]
-    def message_exists?(response)
-      response.respond_to?(:name) && response.name == 'EXISTS'
-    end
-
-    # @private
-    def idle_handler
-      lambda {|response| imap.idle_done if message_exists?(response)}
-    end
-
-    # maintain an imap idle connection
-    def idle
-      return unless ready_to_idle?
-
-      @mailbox.logger.info({ context: @mailbox.context, action: "Idling" })
-      @idling = true
-
-      imap.idle(@mailbox.idle_timeout, &idle_handler)
-    ensure
-      @idling = false
-    end
-
-    # trigger the idle to finish and wait for the thread to finish
-    def stop_idling
-      return unless idling?
-
-      imap.idle_done
-
-      # idling_thread.join
-      # self.idling_thread = nil
-    end
-
-    def process_mailbox
-      return unless @new_message_handler
-      @mailbox.logger.info({ context: @mailbox.context, action: "Processing started" })
-
-      msgs = new_messages
-
-      any_deletions = msgs.
-                      # deliver each new message, collect success
-                      map(&@new_message_handler).
-                      # include messages with success
-                      zip(msgs).
-                      # filter failed deliveries, collect message
-                      select(&:first).map(&:last).
-                      # scrub delivered messages
-                      map { |message| scrub(message) }.
-                      any?
-
-      imap.expunge if @mailbox.expunge_deleted && any_deletions
-    end
-
-    def scrub(message)
-      if @mailbox.delete_after_delivery
-        imap.store(message.seqno, "+FLAGS", [Net::IMAP::DELETED])
-        true
-      end
-    end
-
-    # @private
-    # fetch all messages for the new message ids
-    def new_messages
-      # Both of these calls may results in
-      #   imap raising an EOFError, we handle
-      #   this exception in the watcher
-      messages_for_ids(new_message_ids)
-    end
-
-    # TODO: label messages?
-    #   @imap.store(id, "+X-GM-LABELS", [label])
-
-    # @private
-    # search for all new (unseen) message ids
-    # @return [Array<Integer>] message ids
-    def new_message_ids
-      # uid_search still leaves messages UNSEEN
-      all_unread = @imap.uid_search(@mailbox.search_command)
-
-      to_deliver = all_unread.select { |uid| @mailbox.deliver?(uid) }
-      @mailbox.logger.info({ context: @mailbox.context, action: "Getting new messages", unread: {count: all_unread.count, ids: all_unread}, to_be_delivered: { count: to_deliver.count, ids: all_unread } })
-      to_deliver
-    end
-
-    # @private
-    # fetch the email for all given ids in RFC822 format
-    # @param ids [Array<Integer>] list of message ids
-    # @return [Array<Net::IMAP::FetchData>] the net/imap messages for the given ids
-    def messages_for_ids(uids)
-      return [] if uids.empty?
-
-      # uid_fetch marks as SEEN, will not be re-fetched for UNSEEN
-      imap.uid_fetch(uids, "RFC822")
-    end
+    def quit; end
   end
 end

data/lib/mail_room/crash_handler.rb

@@ -0,0 +1,26 @@
+require 'date'
+
+module MailRoom
+  class CrashHandler
+    SUPPORTED_FORMATS = %w[json none]
+
+    def initialize(stream=STDOUT)
+      @stream = stream
+    end
+
+    def handle(error, format)
+      if format == 'json'
+        @stream.puts json(error)
+        return
+      end
+
+      raise error
+    end
+
+    private
+
+    def json(error)
+      { time: DateTime.now.iso8601(3), severity: :fatal, message: error.message, backtrace: error.backtrace }.to_json
+    end
+  end
+end

data/lib/mail_room/delivery/letter_opener.rb

@@ -24,7 +24,7 @@ module MailRoom
       # Trigger `LetterOpener` to deliver our message
       # @param message [String] the email message as a string, RFC822 format
       def deliver(message)
-        method = ::LetterOpener::DeliveryMethod.new(:location => @delivery_options.location)
+        method = ::LetterOpener::DeliveryMethod.new(location: @delivery_options.location)
         method.deliver!(Mail.read_from_string(message))
 
         true

data/lib/mail_room/delivery/postback.rb

@@ -5,7 +5,7 @@ module MailRoom
     # Postback Delivery method
     # @author Tony Pitale
     class Postback
-      Options = Struct.new(:url, :token, :username, :password, :logger) do
+      Options = Struct.new(:url, :token, :username, :password, :logger, :content_type) do
         def initialize(mailbox)
           url =
             mailbox.delivery_url ||
@@ -22,7 +22,9 @@ module MailRoom
 
           logger = mailbox.logger
 
-          super(url, token, username, password, logger)
+          content_type = mailbox.delivery_options[:content_type]
+
+          super(url, token, username, password, logger, content_type)
         end
 
         def token_auth?
@@ -59,7 +61,7 @@ module MailRoom
           request.url @delivery_options.url
           request.body = message
           # request.options[:timeout] = 3
-          # request.headers['Content-Type'] = 'text/plain'
+          request.headers['Content-Type'] = @delivery_options.content_type unless @delivery_options.content_type.nil?
         end
 
         @delivery_options.logger.info({ delivery_method: 'Postback', action: 'message pushed', url: @delivery_options.url })

data/lib/mail_room/delivery/sidekiq.rb

@@ -8,16 +8,17 @@ module MailRoom
     # Sidekiq Delivery method
     # @author Douwe Maan
     class Sidekiq
-      Options = Struct.new(:redis_url, :namespace, :sentinels, :queue, :worker, :logger) do
+      Options = Struct.new(:redis_url, :namespace, :sentinels, :queue, :worker, :logger, :redis_db) do
         def initialize(mailbox)
           redis_url = mailbox.delivery_options[:redis_url] || "redis://localhost:6379"
+          redis_db  = mailbox.delivery_options[:redis_db] || 0
           namespace = mailbox.delivery_options[:namespace]
           sentinels = mailbox.delivery_options[:sentinels]
           queue     = mailbox.delivery_options[:queue] || "default"
           worker    = mailbox.delivery_options[:worker]
           logger = mailbox.logger
 
-          super(redis_url, namespace, sentinels, queue, worker, logger)
+          super(redis_url, namespace, sentinels, queue, worker, logger, redis_db)
         end
       end
 
@@ -45,7 +46,7 @@ module MailRoom
       def client
         @client ||= begin
           sentinels = options.sentinels
-          redis_options = { url: options.redis_url }
+          redis_options = { url: options.redis_url, db: options.redis_db }
           redis_options[:sentinels] = sentinels if sentinels
 
           redis = ::Redis.new(redis_options)