mail_room 0.9.1 → 0.10.0

This diff has not been reviewed by any users.
Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
- SHA1:
3
- metadata.gz: a31789d70291c209682a4b9ce5af4f5fa894cfba
4
- data.tar.gz: 56f2f340095b95a616bac7a22a1786df517eb688
2
+ SHA256:
3
+ metadata.gz: 50ef3164bc5ce5e15087123671dadb214280376dc4e92cc9bdf62ec3c92a6ca1
4
+ data.tar.gz: 1dd1a45af0fcd31f5b64aac56d6546a7b46eb2d2c0c1cb7e1753fe8d09e57b49
5
5
  SHA512:
6
- metadata.gz: 67fff02a10229d437653d5e02a015fed44d3a184c7f5cc5742cdb08b0c30043b03027eedf1d6b8c36ffca6350eefb143a5a34548f3f2d8998468722c50d7c2b6
7
- data.tar.gz: 696b3b95f5f96fb9a987c75e5cc169c9a346cbf3c9f392777c6fca5691a53045d96514123372c1f58f6767e2734ad442618a7ffc3488864c3b72828ef1c2b3d5
6
+ metadata.gz: a661b81b53b230a51b89a68d54a9711b8f68b7b9ed0ebafc206db9839894abb42f361efe49014ea325b0b528303b5ed9e78c38be7a301b2df91c6efd22565f5c
7
+ data.tar.gz: 97f6e54d35ea41c3935db82b57c932d89869c5dd76e8ba2fd92b5024e41bfc79a26dbaa67caffce3952d93ff3144de9ff0a8d10c78f1c52f2043c15282af3d22
@@ -1 +1 @@
1
- 2.2.2
1
+ 2.6
@@ -1,8 +1,10 @@
1
1
  language: ruby
2
2
  rvm:
3
- - 2.0.0
4
- - 2.1.5
5
- - 2.2.4
6
3
  - 2.3.0
4
+ - 2.4
5
+ - 2.5
6
+ - 2.6
7
+ services:
8
+ - redis-server
7
9
  script: bundle exec rspec spec
8
10
  sudo: false
@@ -1,3 +1,24 @@
1
+ ## mail_room 0.10.0 ##
2
+
3
+ * Remove imap backports
4
+ * Increase minimum ruby version to 2.3
5
+ * Postback basic_auth support - PR#92
6
+ * Docs for ActionMailbox - PR#92
7
+ * Configuration option for delivery_klass - PR#93
8
+ * Expunge deleted - PR#90
9
+ * Raise error on a few fields of missing configuration - PR#89
10
+ * Remove fakeredis gem - PR#87
11
+
12
+ *Tony Pitale <@tpitale>*
13
+
14
+ * Fix redis arbitration to use NX+EX - PR#86
15
+
16
+ *Craig Miskell <@craigmiskell-gitlab>*
17
+
18
+ * Structured (JSON) logger - PR#88
19
+
20
+ *charlie <@cablett>*
21
+
1
22
  ## mail_room 0.9.1 ##
2
23
 
3
24
  * __FILE__ support in yml ERb config - PR#80
data/README.md CHANGED
@@ -41,6 +41,8 @@ You will also need to install `faraday` or `letter_opener` if you use the `postb
41
41
  :password: "password"
42
42
  :name: "inbox"
43
43
  :search_command: 'NEW'
44
+ :logger:
45
+ :log_path: /path/to/logfile/for/mailroom
44
46
  :delivery_options:
45
47
  :delivery_url: "http://localhost:3000/inbox"
46
48
  :delivery_token: "abcdefg"
@@ -66,6 +68,7 @@ You will also need to install `faraday` or `letter_opener` if you use the `postb
66
68
  :name: "inbox"
67
69
  :delivery_method: letter_opener
68
70
  :delete_after_delivery: true
71
+ :expunge_deleted: true
69
72
  :delivery_options:
70
73
  :location: "/tmp/user4-email"
71
74
  -
@@ -92,6 +95,9 @@ You will also need to install `faraday` or `letter_opener` if you use the `postb
92
95
  :worker: EmailReceiverWorker
93
96
  ```
94
97
 
98
+ **Note:** If using `delete_after_delivery`, you also probably want to use
99
+ `expunge_deleted` unless you really know what you're doing.
100
+
95
101
  ## delivery_method ##
96
102
 
97
103
  ### postback ###
@@ -181,7 +187,7 @@ end
181
187
 
182
188
  Configured with `:delivery_method: logger`.
183
189
 
184
- If `:log_path:` is not provided, defaults to `STDOUT`
190
+ If the `:log_path:` delivery option is not provided, defaults to `STDOUT`
185
191
 
186
192
  ### noop ###
187
193
 
@@ -197,6 +203,31 @@ Configured with `:delivery_method: letter_opener`.
197
203
 
198
204
  Uses Ryan Bates' excellent [letter_opener](https://github.com/ryanb/letter_opener) gem.
199
205
 
206
+ ## ActionMailbox in Rails ##
207
+
208
+ MailRoom can deliver mail to Rails using the ActionMailbox [configuration options for an SMTP relay](https://edgeguides.rubyonrails.org/action_mailbox_basics.html#configuration).
209
+
210
+ In summary (from the ActionMailbox docs)
211
+
212
+ 1. Configure Rails to use the `:relay` ingress option:
213
+ ```rb
214
+ # config/environments/production.rb
215
+ config.action_mailbox.ingress = :relay
216
+ ```
217
+
218
+ 2. Generate a strong password (e.g., using SecureRandom or something) and add it to Rails config:
219
+ using `rails credentials:edit` under `action_mailbox.ingress_password`.
220
+
221
+ And finally, configure MailRoom to use the postback configuration with the options:
222
+
223
+ ```yaml
224
+ :delivery_method: postback
225
+ :delivery_options:
226
+ :delivery_url: https://example.com/rails/action_mailbox/relay/inbound_emails
227
+ :delivery_username: actionmailbox
228
+ :delivery_password: <INGRESS_PASSWORD>
229
+ ```
230
+
200
231
  ## Receiving `postback` in Rails ##
201
232
 
202
233
  If you have a controller that you're sending to, with forgery protection
@@ -291,6 +322,12 @@ redis://:<password>@<master-name>/
291
322
  You also have to inform at least one pair of `host` and `port` for a sentinel in your cluster.
292
323
  To have a minimum reliable setup, you need at least `3` sentinel nodes and `3` redis servers (1 master, 2 slaves).
293
324
 
325
+ ## Logging ##
326
+
327
+ MailRoom will output JSON-formatted logs to give some observability into its operations.
328
+
329
+ Simply configure a `log_path` for the `logger` on any of your mailboxes. By default, nothing will be logged.
330
+
294
331
  ## Contributing ##
295
332
 
296
333
  1. Fork it
@@ -6,10 +6,10 @@ module MailRoom
6
6
  end
7
7
 
8
8
  require "mail_room/version"
9
- require "mail_room/backports/imap"
10
9
  require "mail_room/configuration"
11
10
  require "mail_room/mailbox"
12
11
  require "mail_room/mailbox_watcher"
13
12
  require "mail_room/connection"
14
13
  require "mail_room/coordinator"
15
14
  require "mail_room/cli"
15
+ require 'mail_room/logger/structured'
@@ -22,23 +22,16 @@ module MailRoom
22
22
  @options = options
23
23
  end
24
24
 
25
- def deliver?(uid)
25
+ def deliver?(uid, expiration = EXPIRATION)
26
26
  key = "delivered:#{uid}"
27
27
 
28
- incr = nil
29
- client.multi do |c|
30
- # At this point, `incr` is a future, which will get its value after
31
- # the MULTI command returns.
32
- incr = c.incr(key)
33
-
34
- c.expire(key, EXPIRATION)
35
- end
36
-
37
- # If INCR returns 1, that means the key didn't exist before, which means
38
- # we are the first mail_room to try to deliver this message, so we get to.
39
- # If we get any other value, another mail_room already (tried to) deliver
40
- # the message, so we don't have to anymore.
41
- incr.value == 1
28
+ # Set the key, but only if it doesn't already exist;
29
+ # the return value is true if successful, false if the key was already set,
30
+ # which is conveniently the correct return value for this method
31
+ # Any subsequent failure in the instance which gets the lock will be dealt
32
+ # with by the expiration, at which time another instance can pick up the
33
+ # message and try again.
34
+ client.set(key, 1, {:nx => true, :ex => expiration})
42
35
  end
43
36
 
44
37
  private
@@ -50,6 +50,7 @@ module MailRoom
50
50
 
51
51
  process_mailbox
52
52
  rescue Net::IMAP::Error, IOError
53
+ @mailbox.logger.warn({ context: @mailbox.context, action: "Disconnected. Resetting..." })
53
54
  reset
54
55
  setup
55
56
  end
@@ -64,14 +65,19 @@ module MailRoom
64
65
  end
65
66
 
66
67
  def setup
68
+ @mailbox.logger.info({ context: @mailbox.context, action: "Starting TLS session" })
67
69
  start_tls
70
+
71
+ @mailbox.logger.info({ context: @mailbox.context, action: "Logging into mailbox" })
68
72
  log_in
73
+
74
+ @mailbox.logger.info({ context: @mailbox.context, action: "Setting mailbox" })
69
75
  set_mailbox
70
76
  end
71
77
 
72
78
  # build a net/imap connection to google imap
73
79
  def imap
74
- @imap ||= MailRoom::IMAP.new(@mailbox.host, :port => @mailbox.port, :ssl => @mailbox.ssl_options)
80
+ @imap ||= Net::IMAP.new(@mailbox.host, :port => @mailbox.port, :ssl => @mailbox.ssl_options)
75
81
  end
76
82
 
77
83
  # start a TLS session
@@ -106,6 +112,7 @@ module MailRoom
106
112
  def idle
107
113
  return unless ready_to_idle?
108
114
 
115
+ @mailbox.logger.info({ context: @mailbox.context, action: "Idling" })
109
116
  @idling = true
110
117
 
111
118
  imap.idle(@mailbox.idle_timeout, &idle_handler)
@@ -118,26 +125,35 @@ module MailRoom
118
125
  return unless idling?
119
126
 
120
127
  imap.idle_done
121
-
128
+
122
129
  # idling_thread.join
123
130
  # self.idling_thread = nil
124
131
  end
125
132
 
126
133
  def process_mailbox
127
134
  return unless @new_message_handler
135
+ @mailbox.logger.info({ context: @mailbox.context, action: "Processing started" })
128
136
 
129
137
  msgs = new_messages
130
138
 
131
- msgs.
132
- map(&@new_message_handler). # deliver each new message, collect success
133
- zip(msgs). # include messages with success
134
- select(&:first).map(&:last). # filter failed deliveries, collect message
135
- each {|message| scrub(message)} # scrub delivered messages
139
+ any_deletions = msgs.
140
+ # deliver each new message, collect success
141
+ map(&@new_message_handler).
142
+ # include messages with success
143
+ zip(msgs).
144
+ # filter failed deliveries, collect message
145
+ select(&:first).map(&:last).
146
+ # scrub delivered messages
147
+ map { |message| scrub(message) }.
148
+ any?
149
+
150
+ imap.expunge if @mailbox.expunge_deleted && any_deletions
136
151
  end
137
152
 
138
153
  def scrub(message)
139
154
  if @mailbox.delete_after_delivery
140
155
  imap.store(message.seqno, "+FLAGS", [Net::IMAP::DELETED])
156
+ true
141
157
  end
142
158
  end
143
159
 
@@ -158,7 +174,11 @@ module MailRoom
158
174
  # @return [Array<Integer>] message ids
159
175
  def new_message_ids
160
176
  # uid_search still leaves messages UNSEEN
161
- imap.uid_search(@mailbox.search_command).select { |uid| @mailbox.deliver?(uid) }
177
+ all_unread = @imap.uid_search(@mailbox.search_command)
178
+
179
+ to_deliver = all_unread.select { |uid| @mailbox.deliver?(uid) }
180
+ @mailbox.logger.info({ context: @mailbox.context, action: "Getting new messages", unread: {count: all_unread.count, ids: all_unread}, to_be_delivered: { count: to_deliver.count, ids: all_unread } })
181
+ to_deliver
162
182
  end
163
183
 
164
184
  # @private
@@ -5,18 +5,39 @@ module MailRoom
5
5
  # Postback Delivery method
6
6
  # @author Tony Pitale
7
7
  class Postback
8
- Options = Struct.new(:delivery_url, :delivery_token) do
8
+ Options = Struct.new(:url, :token, :username, :password, :logger) do
9
9
  def initialize(mailbox)
10
- delivery_url = mailbox.delivery_url || mailbox.delivery_options[:delivery_url]
11
- delivery_token = mailbox.delivery_token || mailbox.delivery_options[:delivery_token]
10
+ url =
11
+ mailbox.delivery_url ||
12
+ mailbox.delivery_options[:delivery_url] ||
13
+ mailbox.delivery_options[:url]
12
14
 
13
- super(delivery_url, delivery_token)
15
+ token =
16
+ mailbox.delivery_token ||
17
+ mailbox.delivery_options[:delivery_token] ||
18
+ mailbox.delivery_options[:token]
19
+
20
+ username = mailbox.delivery_options[:username]
21
+ password = mailbox.delivery_options[:password]
22
+
23
+ logger = mailbox.logger
24
+
25
+ super(url, token, username, password, logger)
26
+ end
27
+
28
+ def token_auth?
29
+ !self[:token].nil?
30
+ end
31
+
32
+ def basic_auth?
33
+ !self[:username].nil? && !self[:password].nil?
14
34
  end
15
35
  end
16
36
 
17
37
  # Build a new delivery, hold the delivery options
18
38
  # @param [MailRoom::Delivery::Postback::Options]
19
39
  def initialize(delivery_options)
40
+ puts delivery_options
20
41
  @delivery_options = delivery_options
21
42
  end
22
43
 
@@ -24,15 +45,24 @@ module MailRoom
24
45
  # @param message [String] the email message as a string, RFC822 format
25
46
  def deliver(message)
26
47
  connection = Faraday.new
27
- connection.token_auth @delivery_options.delivery_token
48
+
49
+ if @delivery_options.token_auth?
50
+ connection.token_auth @delivery_options.token
51
+ elsif @delivery_options.basic_auth?
52
+ connection.basic_auth(
53
+ @delivery_options.username,
54
+ @delivery_options.password
55
+ )
56
+ end
28
57
 
29
58
  connection.post do |request|
30
- request.url @delivery_options.delivery_url
59
+ request.url @delivery_options.url
31
60
  request.body = message
32
61
  # request.options[:timeout] = 3
33
62
  # request.headers['Content-Type'] = 'text/plain'
34
63
  end
35
64
 
65
+ @delivery_options.logger.info({ delivery_method: 'Postback', action: 'message pushed', url: @delivery_options.url })
36
66
  true
37
67
  end
38
68
  end
@@ -6,7 +6,7 @@ module MailRoom
6
6
  # Que Delivery method
7
7
  # @author Tony Pitale
8
8
  class Que
9
- Options = Struct.new(:host, :port, :database, :username, :password, :queue, :priority, :job_class) do
9
+ Options = Struct.new(:host, :port, :database, :username, :password, :queue, :priority, :job_class, :logger) do
10
10
  def initialize(mailbox)
11
11
  host = mailbox.delivery_options[:host] || "localhost"
12
12
  port = mailbox.delivery_options[:port] || 5432
@@ -17,8 +17,9 @@ module MailRoom
17
17
  queue = mailbox.delivery_options[:queue] || ''
18
18
  priority = mailbox.delivery_options[:priority] || 100 # lowest priority for Que
19
19
  job_class = mailbox.delivery_options[:job_class]
20
+ logger = mailbox.logger
20
21
 
21
- super(host, port, database, username, password, queue, priority, job_class)
22
+ super(host, port, database, username, password, queue, priority, job_class, logger)
22
23
  end
23
24
  end
24
25
 
@@ -34,6 +35,7 @@ module MailRoom
34
35
  # @param message [String] the email message as a string, RFC822 format
35
36
  def deliver(message)
36
37
  queue_job(message)
38
+ @options.logger.info({ delivery_method: 'Que', action: 'message pushed' })
37
39
  end
38
40
 
39
41
  private
@@ -8,15 +8,16 @@ module MailRoom
8
8
  # Sidekiq Delivery method
9
9
  # @author Douwe Maan
10
10
  class Sidekiq
11
- Options = Struct.new(:redis_url, :namespace, :sentinels, :queue, :worker) do
11
+ Options = Struct.new(:redis_url, :namespace, :sentinels, :queue, :worker, :logger) do
12
12
  def initialize(mailbox)
13
13
  redis_url = mailbox.delivery_options[:redis_url] || "redis://localhost:6379"
14
14
  namespace = mailbox.delivery_options[:namespace]
15
15
  sentinels = mailbox.delivery_options[:sentinels]
16
16
  queue = mailbox.delivery_options[:queue] || "default"
17
17
  worker = mailbox.delivery_options[:worker]
18
+ logger = mailbox.logger
18
19
 
19
- super(redis_url, namespace, sentinels, queue, worker)
20
+ super(redis_url, namespace, sentinels, queue, worker, logger)
20
21
  end
21
22
  end
22
23
 
@@ -35,6 +36,7 @@ module MailRoom
35
36
 
36
37
  client.lpush("queue:#{options.queue}", JSON.generate(item))
37
38
 
39
+ @options.logger.info({ delivery_method: 'Sidekiq', action: 'message pushed' })
38
40
  true
39
41
  end
40
42
 
@@ -62,7 +64,6 @@ module MailRoom
62
64
  {
63
65
  'class' => options.worker,
64
66
  'args' => [utf8_encode_message(message)],
65
-
66
67
  'queue' => options.queue,
67
68
  'jid' => SecureRandom.hex(12),
68
69
  'retry' => false,
@@ -0,0 +1,21 @@
1
+ require 'logger'
2
+ require 'json'
3
+
4
+ module MailRoom
5
+ module Logger
6
+ class Structured < ::Logger
7
+
8
+ def format_message(severity, timestamp, progname, message)
9
+ raise ArgumentError.new("Message must be a Hash") unless message.is_a? Hash
10
+
11
+ data = {}
12
+ data[:severity] = severity
13
+ data[:time] = timestamp || Time.now.to_s
14
+ # only accept a Hash
15
+ data.merge!(message)
16
+
17
+ data.to_json + "\n"
18
+ end
19
+ end
20
+ end
21
+ end
@@ -14,6 +14,8 @@ module MailRoom
14
14
  :search_command,
15
15
  :name,
16
16
  :delete_after_delivery,
17
+ :expunge_deleted,
18
+ :delivery_klass,
17
19
  :delivery_method, # :noop, :logger, :postback, :letter_opener
18
20
  :log_path, # for logger
19
21
  :delivery_url, # for postback
@@ -21,9 +23,11 @@ module MailRoom
21
23
  :location, # for letter_opener
22
24
  :delivery_options,
23
25
  :arbitration_method,
24
- :arbitration_options
26
+ :arbitration_options,
27
+ :logger
25
28
  ]
26
29
 
30
+ ConfigurationError = Class.new(RuntimeError)
27
31
  IdleTimeoutTooLarge = Class.new(RuntimeError)
28
32
 
29
33
  # Holds configuration for each of the email accounts we wish to monitor
@@ -35,6 +39,8 @@ module MailRoom
35
39
  # 29 minutes, as suggested by the spec: https://tools.ietf.org/html/rfc2177
36
40
  IMAP_IDLE_TIMEOUT = 29 * 60 # 29 minutes in in seconds
37
41
 
42
+ REQUIRED_CONFIGURATION = [:name, :email, :password, :host, :port]
43
+
38
44
  # Default attributes for the mailbox configuration
39
45
  DEFAULTS = {
40
46
  :search_command => 'UNSEEN',
@@ -45,9 +51,11 @@ module MailRoom
45
51
  :start_tls => false,
46
52
  :idle_timeout => IMAP_IDLE_TIMEOUT,
47
53
  :delete_after_delivery => false,
54
+ :expunge_deleted => false,
48
55
  :delivery_options => {},
49
56
  :arbitration_method => 'noop',
50
- :arbitration_options => {}
57
+ :arbitration_options => {},
58
+ :logger => {}
51
59
  }
52
60
 
53
61
  # Store the configuration and require the appropriate delivery method
@@ -58,8 +66,19 @@ module MailRoom
58
66
  validate!
59
67
  end
60
68
 
69
+ def logger
70
+ @logger ||=
71
+ case self[:logger]
72
+ when Logger
73
+ self[:logger]
74
+ else
75
+ self[:logger] ||= {}
76
+ MailRoom::Logger::Structured.new(self[:logger][:log_path])
77
+ end
78
+ end
79
+
61
80
  def delivery_klass
62
- Delivery[delivery_method]
81
+ self[:delivery_klass] ||= Delivery[delivery_method]
63
82
  end
64
83
 
65
84
  def arbitration_klass
@@ -75,6 +94,8 @@ module MailRoom
75
94
  end
76
95
 
77
96
  def deliver?(uid)
97
+ logger.info({context: context, uid: uid, action: "asking arbiter to deliver", arbitrator: arbitrator.class.name})
98
+
78
99
  arbitrator.deliver?(uid)
79
100
  end
80
101
 
@@ -84,6 +105,7 @@ module MailRoom
84
105
  body = message.attr['RFC822']
85
106
  return true unless body
86
107
 
108
+ logger.info({context: context, uid: message.attr['UID'], action: "sending to deliverer", deliverer: delivery.class.name, byte_size: message.attr['RFC822.SIZE']})
87
109
  delivery.deliver(body)
88
110
  end
89
111
 
@@ -92,9 +114,22 @@ module MailRoom
92
114
  replace_verify_mode(ssl)
93
115
  end
94
116
 
117
+ def context
118
+ { email: self.email, name: self.name }
119
+ end
120
+
95
121
  def validate!
96
122
  if self[:idle_timeout] > IMAP_IDLE_TIMEOUT
97
- raise IdleTimeoutTooLarge.new("Please use an idle timeout smaller than #{29*60} to prevent IMAP server disconnects")
123
+ raise IdleTimeoutTooLarge,
124
+ "Please use an idle timeout smaller than #{29*60} to prevent " \
125
+ "IMAP server disconnects"
126
+ end
127
+
128
+ REQUIRED_CONFIGURATION.each do |k|
129
+ if self[k].nil?
130
+ raise ConfigurationError,
131
+ "Field :#{k} is required in Mailbox: #{inspect}"
132
+ end
98
133
  end
99
134
  end
100
135
 
@@ -112,18 +147,22 @@ module MailRoom
112
147
  return options unless options.is_a?(Hash)
113
148
  return options unless options.has_key?(:verify_mode)
114
149
 
115
- options[:verify_mode] = case options[:verify_mode]
116
- when :none, 'none'
117
- OpenSSL::SSL::VERIFY_NONE
118
- when :peer, 'peer'
119
- OpenSSL::SSL::VERIFY_PEER
120
- when :client_once, 'client_once'
121
- OpenSSL::SSL::VERIFY_CLIENT_ONCE
122
- when :fail_if_no_peer_cert, 'fail_if_no_peer_cert'
123
- OpenSSL::SSL::VERIFY_FAIL_IF_NO_PEER_CERT
124
- end
150
+ options[:verify_mode] = lookup_verify_mode(options[:verify_mode])
125
151
 
126
152
  options
127
153
  end
154
+
155
+ def lookup_verify_mode(verify_mode)
156
+ case verify_mode.to_sym
157
+ when :none
158
+ OpenSSL::SSL::VERIFY_NONE
159
+ when :peer
160
+ OpenSSL::SSL::VERIFY_PEER
161
+ when :client_once
162
+ OpenSSL::SSL::VERIFY_CLIENT_ONCE
163
+ when :fail_if_no_peer_cert
164
+ OpenSSL::SSL::VERIFY_FAIL_IF_NO_PEER_CERT
165
+ end
166
+ end
128
167
  end
129
168
  end