mail_room 0.9.1 → 0.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
- SHA1:
3
- metadata.gz: a31789d70291c209682a4b9ce5af4f5fa894cfba
4
- data.tar.gz: 56f2f340095b95a616bac7a22a1786df517eb688
2
+ SHA256:
3
+ metadata.gz: 50ef3164bc5ce5e15087123671dadb214280376dc4e92cc9bdf62ec3c92a6ca1
4
+ data.tar.gz: 1dd1a45af0fcd31f5b64aac56d6546a7b46eb2d2c0c1cb7e1753fe8d09e57b49
5
5
  SHA512:
6
- metadata.gz: 67fff02a10229d437653d5e02a015fed44d3a184c7f5cc5742cdb08b0c30043b03027eedf1d6b8c36ffca6350eefb143a5a34548f3f2d8998468722c50d7c2b6
7
- data.tar.gz: 696b3b95f5f96fb9a987c75e5cc169c9a346cbf3c9f392777c6fca5691a53045d96514123372c1f58f6767e2734ad442618a7ffc3488864c3b72828ef1c2b3d5
6
+ metadata.gz: a661b81b53b230a51b89a68d54a9711b8f68b7b9ed0ebafc206db9839894abb42f361efe49014ea325b0b528303b5ed9e78c38be7a301b2df91c6efd22565f5c
7
+ data.tar.gz: 97f6e54d35ea41c3935db82b57c932d89869c5dd76e8ba2fd92b5024e41bfc79a26dbaa67caffce3952d93ff3144de9ff0a8d10c78f1c52f2043c15282af3d22
@@ -1 +1 @@
1
- 2.2.2
1
+ 2.6
@@ -1,8 +1,10 @@
1
1
  language: ruby
2
2
  rvm:
3
- - 2.0.0
4
- - 2.1.5
5
- - 2.2.4
6
3
  - 2.3.0
4
+ - 2.4
5
+ - 2.5
6
+ - 2.6
7
+ services:
8
+ - redis-server
7
9
  script: bundle exec rspec spec
8
10
  sudo: false
@@ -1,3 +1,24 @@
1
+ ## mail_room 0.10.0 ##
2
+
3
+ * Remove imap backports
4
+ * Increase minimum ruby version to 2.3
5
+ * Postback basic_auth support - PR#92
6
+ * Docs for ActionMailbox - PR#92
7
+ * Configuration option for delivery_klass - PR#93
8
+ * Expunge deleted - PR#90
9
+ * Raise error on a few fields of missing configuration - PR#89
10
+ * Remove fakeredis gem - PR#87
11
+
12
+ *Tony Pitale <@tpitale>*
13
+
14
+ * Fix redis arbitration to use NX+EX - PR#86
15
+
16
+ *Craig Miskell <@craigmiskell-gitlab>*
17
+
18
+ * Structured (JSON) logger - PR#88
19
+
20
+ *charlie <@cablett>*
21
+
1
22
  ## mail_room 0.9.1 ##
2
23
 
3
24
  * __FILE__ support in yml ERb config - PR#80
data/README.md CHANGED
@@ -41,6 +41,8 @@ You will also need to install `faraday` or `letter_opener` if you use the `postb
41
41
  :password: "password"
42
42
  :name: "inbox"
43
43
  :search_command: 'NEW'
44
+ :logger:
45
+ :log_path: /path/to/logfile/for/mailroom
44
46
  :delivery_options:
45
47
  :delivery_url: "http://localhost:3000/inbox"
46
48
  :delivery_token: "abcdefg"
@@ -66,6 +68,7 @@ You will also need to install `faraday` or `letter_opener` if you use the `postb
66
68
  :name: "inbox"
67
69
  :delivery_method: letter_opener
68
70
  :delete_after_delivery: true
71
+ :expunge_deleted: true
69
72
  :delivery_options:
70
73
  :location: "/tmp/user4-email"
71
74
  -
@@ -92,6 +95,9 @@ You will also need to install `faraday` or `letter_opener` if you use the `postb
92
95
  :worker: EmailReceiverWorker
93
96
  ```
94
97
 
98
+ **Note:** If using `delete_after_delivery`, you also probably want to use
99
+ `expunge_deleted` unless you really know what you're doing.
100
+
95
101
  ## delivery_method ##
96
102
 
97
103
  ### postback ###
@@ -181,7 +187,7 @@ end
181
187
 
182
188
  Configured with `:delivery_method: logger`.
183
189
 
184
- If `:log_path:` is not provided, defaults to `STDOUT`
190
+ If the `:log_path:` delivery option is not provided, defaults to `STDOUT`
185
191
 
186
192
  ### noop ###
187
193
 
@@ -197,6 +203,31 @@ Configured with `:delivery_method: letter_opener`.
197
203
 
198
204
  Uses Ryan Bates' excellent [letter_opener](https://github.com/ryanb/letter_opener) gem.
199
205
 
206
+ ## ActionMailbox in Rails ##
207
+
208
+ MailRoom can deliver mail to Rails using the ActionMailbox [configuration options for an SMTP relay](https://edgeguides.rubyonrails.org/action_mailbox_basics.html#configuration).
209
+
210
+ In summary (from the ActionMailbox docs)
211
+
212
+ 1. Configure Rails to use the `:relay` ingress option:
213
+ ```rb
214
+ # config/environments/production.rb
215
+ config.action_mailbox.ingress = :relay
216
+ ```
217
+
218
+ 2. Generate a strong password (e.g., using SecureRandom or something) and add it to Rails config:
219
+ using `rails credentials:edit` under `action_mailbox.ingress_password`.
220
+
221
+ And finally, configure MailRoom to use the postback configuration with the options:
222
+
223
+ ```yaml
224
+ :delivery_method: postback
225
+ :delivery_options:
226
+ :delivery_url: https://example.com/rails/action_mailbox/relay/inbound_emails
227
+ :delivery_username: actionmailbox
228
+ :delivery_password: <INGRESS_PASSWORD>
229
+ ```
230
+
200
231
  ## Receiving `postback` in Rails ##
201
232
 
202
233
  If you have a controller that you're sending to, with forgery protection
@@ -291,6 +322,12 @@ redis://:<password>@<master-name>/
291
322
  You also have to inform at least one pair of `host` and `port` for a sentinel in your cluster.
292
323
  To have a minimum reliable setup, you need at least `3` sentinel nodes and `3` redis servers (1 master, 2 slaves).
293
324
 
325
+ ## Logging ##
326
+
327
+ MailRoom will output JSON-formatted logs to give some observability into its operations.
328
+
329
+ Simply configure a `log_path` for the `logger` on any of your mailboxes. By default, nothing will be logged.
330
+
294
331
  ## Contributing ##
295
332
 
296
333
  1. Fork it
@@ -6,10 +6,10 @@ module MailRoom
6
6
  end
7
7
 
8
8
  require "mail_room/version"
9
- require "mail_room/backports/imap"
10
9
  require "mail_room/configuration"
11
10
  require "mail_room/mailbox"
12
11
  require "mail_room/mailbox_watcher"
13
12
  require "mail_room/connection"
14
13
  require "mail_room/coordinator"
15
14
  require "mail_room/cli"
15
+ require 'mail_room/logger/structured'
@@ -22,23 +22,16 @@ module MailRoom
22
22
  @options = options
23
23
  end
24
24
 
25
- def deliver?(uid)
25
+ def deliver?(uid, expiration = EXPIRATION)
26
26
  key = "delivered:#{uid}"
27
27
 
28
- incr = nil
29
- client.multi do |c|
30
- # At this point, `incr` is a future, which will get its value after
31
- # the MULTI command returns.
32
- incr = c.incr(key)
33
-
34
- c.expire(key, EXPIRATION)
35
- end
36
-
37
- # If INCR returns 1, that means the key didn't exist before, which means
38
- # we are the first mail_room to try to deliver this message, so we get to.
39
- # If we get any other value, another mail_room already (tried to) deliver
40
- # the message, so we don't have to anymore.
41
- incr.value == 1
28
+ # Set the key, but only if it doesn't already exist;
29
+ # the return value is true if successful, false if the key was already set,
30
+ # which is conveniently the correct return value for this method
31
+ # Any subsequent failure in the instance which gets the lock will be dealt
32
+ # with by the expiration, at which time another instance can pick up the
33
+ # message and try again.
34
+ client.set(key, 1, {:nx => true, :ex => expiration})
42
35
  end
43
36
 
44
37
  private
@@ -50,6 +50,7 @@ module MailRoom
50
50
 
51
51
  process_mailbox
52
52
  rescue Net::IMAP::Error, IOError
53
+ @mailbox.logger.warn({ context: @mailbox.context, action: "Disconnected. Resetting..." })
53
54
  reset
54
55
  setup
55
56
  end
@@ -64,14 +65,19 @@ module MailRoom
64
65
  end
65
66
 
66
67
  def setup
68
+ @mailbox.logger.info({ context: @mailbox.context, action: "Starting TLS session" })
67
69
  start_tls
70
+
71
+ @mailbox.logger.info({ context: @mailbox.context, action: "Logging into mailbox" })
68
72
  log_in
73
+
74
+ @mailbox.logger.info({ context: @mailbox.context, action: "Setting mailbox" })
69
75
  set_mailbox
70
76
  end
71
77
 
72
78
  # build a net/imap connection to google imap
73
79
  def imap
74
- @imap ||= MailRoom::IMAP.new(@mailbox.host, :port => @mailbox.port, :ssl => @mailbox.ssl_options)
80
+ @imap ||= Net::IMAP.new(@mailbox.host, :port => @mailbox.port, :ssl => @mailbox.ssl_options)
75
81
  end
76
82
 
77
83
  # start a TLS session
@@ -106,6 +112,7 @@ module MailRoom
106
112
  def idle
107
113
  return unless ready_to_idle?
108
114
 
115
+ @mailbox.logger.info({ context: @mailbox.context, action: "Idling" })
109
116
  @idling = true
110
117
 
111
118
  imap.idle(@mailbox.idle_timeout, &idle_handler)
@@ -118,26 +125,35 @@ module MailRoom
118
125
  return unless idling?
119
126
 
120
127
  imap.idle_done
121
-
128
+
122
129
  # idling_thread.join
123
130
  # self.idling_thread = nil
124
131
  end
125
132
 
126
133
  def process_mailbox
127
134
  return unless @new_message_handler
135
+ @mailbox.logger.info({ context: @mailbox.context, action: "Processing started" })
128
136
 
129
137
  msgs = new_messages
130
138
 
131
- msgs.
132
- map(&@new_message_handler). # deliver each new message, collect success
133
- zip(msgs). # include messages with success
134
- select(&:first).map(&:last). # filter failed deliveries, collect message
135
- each {|message| scrub(message)} # scrub delivered messages
139
+ any_deletions = msgs.
140
+ # deliver each new message, collect success
141
+ map(&@new_message_handler).
142
+ # include messages with success
143
+ zip(msgs).
144
+ # filter failed deliveries, collect message
145
+ select(&:first).map(&:last).
146
+ # scrub delivered messages
147
+ map { |message| scrub(message) }.
148
+ any?
149
+
150
+ imap.expunge if @mailbox.expunge_deleted && any_deletions
136
151
  end
137
152
 
138
153
  def scrub(message)
139
154
  if @mailbox.delete_after_delivery
140
155
  imap.store(message.seqno, "+FLAGS", [Net::IMAP::DELETED])
156
+ true
141
157
  end
142
158
  end
143
159
 
@@ -158,7 +174,11 @@ module MailRoom
158
174
  # @return [Array<Integer>] message ids
159
175
  def new_message_ids
160
176
  # uid_search still leaves messages UNSEEN
161
- imap.uid_search(@mailbox.search_command).select { |uid| @mailbox.deliver?(uid) }
177
+ all_unread = @imap.uid_search(@mailbox.search_command)
178
+
179
+ to_deliver = all_unread.select { |uid| @mailbox.deliver?(uid) }
180
+ @mailbox.logger.info({ context: @mailbox.context, action: "Getting new messages", unread: {count: all_unread.count, ids: all_unread}, to_be_delivered: { count: to_deliver.count, ids: all_unread } })
181
+ to_deliver
162
182
  end
163
183
 
164
184
  # @private
@@ -5,18 +5,39 @@ module MailRoom
5
5
  # Postback Delivery method
6
6
  # @author Tony Pitale
7
7
  class Postback
8
- Options = Struct.new(:delivery_url, :delivery_token) do
8
+ Options = Struct.new(:url, :token, :username, :password, :logger) do
9
9
  def initialize(mailbox)
10
- delivery_url = mailbox.delivery_url || mailbox.delivery_options[:delivery_url]
11
- delivery_token = mailbox.delivery_token || mailbox.delivery_options[:delivery_token]
10
+ url =
11
+ mailbox.delivery_url ||
12
+ mailbox.delivery_options[:delivery_url] ||
13
+ mailbox.delivery_options[:url]
12
14
 
13
- super(delivery_url, delivery_token)
15
+ token =
16
+ mailbox.delivery_token ||
17
+ mailbox.delivery_options[:delivery_token] ||
18
+ mailbox.delivery_options[:token]
19
+
20
+ username = mailbox.delivery_options[:username]
21
+ password = mailbox.delivery_options[:password]
22
+
23
+ logger = mailbox.logger
24
+
25
+ super(url, token, username, password, logger)
26
+ end
27
+
28
+ def token_auth?
29
+ !self[:token].nil?
30
+ end
31
+
32
+ def basic_auth?
33
+ !self[:username].nil? && !self[:password].nil?
14
34
  end
15
35
  end
16
36
 
17
37
  # Build a new delivery, hold the delivery options
18
38
  # @param [MailRoom::Delivery::Postback::Options]
19
39
  def initialize(delivery_options)
40
+ puts delivery_options
20
41
  @delivery_options = delivery_options
21
42
  end
22
43
 
@@ -24,15 +45,24 @@ module MailRoom
24
45
  # @param message [String] the email message as a string, RFC822 format
25
46
  def deliver(message)
26
47
  connection = Faraday.new
27
- connection.token_auth @delivery_options.delivery_token
48
+
49
+ if @delivery_options.token_auth?
50
+ connection.token_auth @delivery_options.token
51
+ elsif @delivery_options.basic_auth?
52
+ connection.basic_auth(
53
+ @delivery_options.username,
54
+ @delivery_options.password
55
+ )
56
+ end
28
57
 
29
58
  connection.post do |request|
30
- request.url @delivery_options.delivery_url
59
+ request.url @delivery_options.url
31
60
  request.body = message
32
61
  # request.options[:timeout] = 3
33
62
  # request.headers['Content-Type'] = 'text/plain'
34
63
  end
35
64
 
65
+ @delivery_options.logger.info({ delivery_method: 'Postback', action: 'message pushed', url: @delivery_options.url })
36
66
  true
37
67
  end
38
68
  end
@@ -6,7 +6,7 @@ module MailRoom
6
6
  # Que Delivery method
7
7
  # @author Tony Pitale
8
8
  class Que
9
- Options = Struct.new(:host, :port, :database, :username, :password, :queue, :priority, :job_class) do
9
+ Options = Struct.new(:host, :port, :database, :username, :password, :queue, :priority, :job_class, :logger) do
10
10
  def initialize(mailbox)
11
11
  host = mailbox.delivery_options[:host] || "localhost"
12
12
  port = mailbox.delivery_options[:port] || 5432
@@ -17,8 +17,9 @@ module MailRoom
17
17
  queue = mailbox.delivery_options[:queue] || ''
18
18
  priority = mailbox.delivery_options[:priority] || 100 # lowest priority for Que
19
19
  job_class = mailbox.delivery_options[:job_class]
20
+ logger = mailbox.logger
20
21
 
21
- super(host, port, database, username, password, queue, priority, job_class)
22
+ super(host, port, database, username, password, queue, priority, job_class, logger)
22
23
  end
23
24
  end
24
25
 
@@ -34,6 +35,7 @@ module MailRoom
34
35
  # @param message [String] the email message as a string, RFC822 format
35
36
  def deliver(message)
36
37
  queue_job(message)
38
+ @options.logger.info({ delivery_method: 'Que', action: 'message pushed' })
37
39
  end
38
40
 
39
41
  private
@@ -8,15 +8,16 @@ module MailRoom
8
8
  # Sidekiq Delivery method
9
9
  # @author Douwe Maan
10
10
  class Sidekiq
11
- Options = Struct.new(:redis_url, :namespace, :sentinels, :queue, :worker) do
11
+ Options = Struct.new(:redis_url, :namespace, :sentinels, :queue, :worker, :logger) do
12
12
  def initialize(mailbox)
13
13
  redis_url = mailbox.delivery_options[:redis_url] || "redis://localhost:6379"
14
14
  namespace = mailbox.delivery_options[:namespace]
15
15
  sentinels = mailbox.delivery_options[:sentinels]
16
16
  queue = mailbox.delivery_options[:queue] || "default"
17
17
  worker = mailbox.delivery_options[:worker]
18
+ logger = mailbox.logger
18
19
 
19
- super(redis_url, namespace, sentinels, queue, worker)
20
+ super(redis_url, namespace, sentinels, queue, worker, logger)
20
21
  end
21
22
  end
22
23
 
@@ -35,6 +36,7 @@ module MailRoom
35
36
 
36
37
  client.lpush("queue:#{options.queue}", JSON.generate(item))
37
38
 
39
+ @options.logger.info({ delivery_method: 'Sidekiq', action: 'message pushed' })
38
40
  true
39
41
  end
40
42
 
@@ -62,7 +64,6 @@ module MailRoom
62
64
  {
63
65
  'class' => options.worker,
64
66
  'args' => [utf8_encode_message(message)],
65
-
66
67
  'queue' => options.queue,
67
68
  'jid' => SecureRandom.hex(12),
68
69
  'retry' => false,
@@ -0,0 +1,21 @@
1
+ require 'logger'
2
+ require 'json'
3
+
4
+ module MailRoom
5
+ module Logger
6
+ class Structured < ::Logger
7
+
8
+ def format_message(severity, timestamp, progname, message)
9
+ raise ArgumentError.new("Message must be a Hash") unless message.is_a? Hash
10
+
11
+ data = {}
12
+ data[:severity] = severity
13
+ data[:time] = timestamp || Time.now.to_s
14
+ # only accept a Hash
15
+ data.merge!(message)
16
+
17
+ data.to_json + "\n"
18
+ end
19
+ end
20
+ end
21
+ end
@@ -14,6 +14,8 @@ module MailRoom
14
14
  :search_command,
15
15
  :name,
16
16
  :delete_after_delivery,
17
+ :expunge_deleted,
18
+ :delivery_klass,
17
19
  :delivery_method, # :noop, :logger, :postback, :letter_opener
18
20
  :log_path, # for logger
19
21
  :delivery_url, # for postback
@@ -21,9 +23,11 @@ module MailRoom
21
23
  :location, # for letter_opener
22
24
  :delivery_options,
23
25
  :arbitration_method,
24
- :arbitration_options
26
+ :arbitration_options,
27
+ :logger
25
28
  ]
26
29
 
30
+ ConfigurationError = Class.new(RuntimeError)
27
31
  IdleTimeoutTooLarge = Class.new(RuntimeError)
28
32
 
29
33
  # Holds configuration for each of the email accounts we wish to monitor
@@ -35,6 +39,8 @@ module MailRoom
35
39
  # 29 minutes, as suggested by the spec: https://tools.ietf.org/html/rfc2177
36
40
  IMAP_IDLE_TIMEOUT = 29 * 60 # 29 minutes in in seconds
37
41
 
42
+ REQUIRED_CONFIGURATION = [:name, :email, :password, :host, :port]
43
+
38
44
  # Default attributes for the mailbox configuration
39
45
  DEFAULTS = {
40
46
  :search_command => 'UNSEEN',
@@ -45,9 +51,11 @@ module MailRoom
45
51
  :start_tls => false,
46
52
  :idle_timeout => IMAP_IDLE_TIMEOUT,
47
53
  :delete_after_delivery => false,
54
+ :expunge_deleted => false,
48
55
  :delivery_options => {},
49
56
  :arbitration_method => 'noop',
50
- :arbitration_options => {}
57
+ :arbitration_options => {},
58
+ :logger => {}
51
59
  }
52
60
 
53
61
  # Store the configuration and require the appropriate delivery method
@@ -58,8 +66,19 @@ module MailRoom
58
66
  validate!
59
67
  end
60
68
 
69
+ def logger
70
+ @logger ||=
71
+ case self[:logger]
72
+ when Logger
73
+ self[:logger]
74
+ else
75
+ self[:logger] ||= {}
76
+ MailRoom::Logger::Structured.new(self[:logger][:log_path])
77
+ end
78
+ end
79
+
61
80
  def delivery_klass
62
- Delivery[delivery_method]
81
+ self[:delivery_klass] ||= Delivery[delivery_method]
63
82
  end
64
83
 
65
84
  def arbitration_klass
@@ -75,6 +94,8 @@ module MailRoom
75
94
  end
76
95
 
77
96
  def deliver?(uid)
97
+ logger.info({context: context, uid: uid, action: "asking arbiter to deliver", arbitrator: arbitrator.class.name})
98
+
78
99
  arbitrator.deliver?(uid)
79
100
  end
80
101
 
@@ -84,6 +105,7 @@ module MailRoom
84
105
  body = message.attr['RFC822']
85
106
  return true unless body
86
107
 
108
+ logger.info({context: context, uid: message.attr['UID'], action: "sending to deliverer", deliverer: delivery.class.name, byte_size: message.attr['RFC822.SIZE']})
87
109
  delivery.deliver(body)
88
110
  end
89
111
 
@@ -92,9 +114,22 @@ module MailRoom
92
114
  replace_verify_mode(ssl)
93
115
  end
94
116
 
117
+ def context
118
+ { email: self.email, name: self.name }
119
+ end
120
+
95
121
  def validate!
96
122
  if self[:idle_timeout] > IMAP_IDLE_TIMEOUT
97
- raise IdleTimeoutTooLarge.new("Please use an idle timeout smaller than #{29*60} to prevent IMAP server disconnects")
123
+ raise IdleTimeoutTooLarge,
124
+ "Please use an idle timeout smaller than #{29*60} to prevent " \
125
+ "IMAP server disconnects"
126
+ end
127
+
128
+ REQUIRED_CONFIGURATION.each do |k|
129
+ if self[k].nil?
130
+ raise ConfigurationError,
131
+ "Field :#{k} is required in Mailbox: #{inspect}"
132
+ end
98
133
  end
99
134
  end
100
135
 
@@ -112,18 +147,22 @@ module MailRoom
112
147
  return options unless options.is_a?(Hash)
113
148
  return options unless options.has_key?(:verify_mode)
114
149
 
115
- options[:verify_mode] = case options[:verify_mode]
116
- when :none, 'none'
117
- OpenSSL::SSL::VERIFY_NONE
118
- when :peer, 'peer'
119
- OpenSSL::SSL::VERIFY_PEER
120
- when :client_once, 'client_once'
121
- OpenSSL::SSL::VERIFY_CLIENT_ONCE
122
- when :fail_if_no_peer_cert, 'fail_if_no_peer_cert'
123
- OpenSSL::SSL::VERIFY_FAIL_IF_NO_PEER_CERT
124
- end
150
+ options[:verify_mode] = lookup_verify_mode(options[:verify_mode])
125
151
 
126
152
  options
127
153
  end
154
+
155
+ def lookup_verify_mode(verify_mode)
156
+ case verify_mode.to_sym
157
+ when :none
158
+ OpenSSL::SSL::VERIFY_NONE
159
+ when :peer
160
+ OpenSSL::SSL::VERIFY_PEER
161
+ when :client_once
162
+ OpenSSL::SSL::VERIFY_CLIENT_ONCE
163
+ when :fail_if_no_peer_cert
164
+ OpenSSL::SSL::VERIFY_FAIL_IF_NO_PEER_CERT
165
+ end
166
+ end
128
167
  end
129
168
  end