gitlab-mail_room 0.0.7 → 0.0.12

Sign up to get free protection for your applications and to get access to all the features.
Files changed (45) hide show
  1. checksums.yaml +4 -4
  2. data/.gitlab-ci.yml +7 -7
  3. data/.rubocop.yml +5 -0
  4. data/.rubocop_todo.yml +501 -0
  5. data/.ruby-version +1 -1
  6. data/.travis.yml +12 -5
  7. data/README.md +104 -10
  8. data/Rakefile +1 -1
  9. data/lib/mail_room.rb +2 -0
  10. data/lib/mail_room/arbitration/redis.rb +1 -1
  11. data/lib/mail_room/cli.rb +1 -1
  12. data/lib/mail_room/configuration.rb +11 -1
  13. data/lib/mail_room/connection.rb +6 -182
  14. data/lib/mail_room/coordinator.rb +8 -4
  15. data/lib/mail_room/crash_handler.rb +2 -2
  16. data/lib/mail_room/delivery/letter_opener.rb +1 -1
  17. data/lib/mail_room/health_check.rb +60 -0
  18. data/lib/mail_room/imap.rb +8 -0
  19. data/lib/mail_room/imap/connection.rb +200 -0
  20. data/lib/mail_room/imap/message.rb +19 -0
  21. data/lib/mail_room/logger/structured.rb +15 -1
  22. data/lib/mail_room/mailbox.rb +61 -21
  23. data/lib/mail_room/mailbox_watcher.rb +15 -2
  24. data/lib/mail_room/message.rb +16 -0
  25. data/lib/mail_room/microsoft_graph.rb +7 -0
  26. data/lib/mail_room/microsoft_graph/connection.rb +217 -0
  27. data/lib/mail_room/version.rb +1 -1
  28. data/mail_room.gemspec +6 -0
  29. data/spec/fixtures/test_config.yml +3 -0
  30. data/spec/lib/cli_spec.rb +6 -6
  31. data/spec/lib/configuration_spec.rb +10 -2
  32. data/spec/lib/coordinator_spec.rb +16 -2
  33. data/spec/lib/delivery/letter_opener_spec.rb +2 -2
  34. data/spec/lib/delivery/logger_spec.rb +1 -1
  35. data/spec/lib/delivery/postback_spec.rb +11 -11
  36. data/spec/lib/health_check_spec.rb +57 -0
  37. data/spec/lib/{connection_spec.rb → imap/connection_spec.rb} +12 -8
  38. data/spec/lib/imap/message_spec.rb +36 -0
  39. data/spec/lib/logger/structured_spec.rb +34 -2
  40. data/spec/lib/mailbox_spec.rb +75 -27
  41. data/spec/lib/mailbox_watcher_spec.rb +54 -38
  42. data/spec/lib/message_spec.rb +35 -0
  43. data/spec/lib/microsoft_graph/connection_spec.rb +190 -0
  44. data/spec/spec_helper.rb +14 -3
  45. metadata +92 -5
data/.ruby-version CHANGED
@@ -1 +1 @@
1
- 2.6.0
1
+ 2.7.2
data/.travis.yml CHANGED
@@ -1,10 +1,17 @@
1
1
  language: ruby
2
2
  rvm:
3
- - 2.3.0
4
- - 2.4
5
3
  - 2.5
6
4
  - 2.6
5
+ - 2.7
6
+ - 3.0
7
+ - truffleruby
7
8
  services:
8
- - redis-server
9
- script: bundle exec rspec spec
10
- sudo: false
9
+ - redis
10
+ script:
11
+ - bundle exec rspec spec
12
+
13
+ jobs:
14
+ - language: ruby
15
+ rvm: 2.7
16
+ script:
17
+ - bundle exec rubocop
data/README.md CHANGED
@@ -20,7 +20,14 @@ The fork is useful as we can post quick fixes to our own fork and release fixes
20
20
 
21
21
  ## README
22
22
 
23
- mail_room is a configuration based process that will idle on IMAP connections and execute a delivery method when a new message is received. Examples of delivery methods include:
23
+ mail_room is a configuration based process that will listen for incoming
24
+ e-mail and execute a delivery method when a new message is
25
+ received. mail_room supports the following methods for receiving e-mail:
26
+
27
+ * IMAP
28
+ * [Microsoft Graph API](https://docs.microsoft.com/en-us/graph/api/resources/mail-api-overview?view=graph-rest-1.0)
29
+
30
+ Examples of delivery methods include:
24
31
 
25
32
  * POST to a delivery URL (Postback)
26
33
  * Queue a job to Sidekiq or Que for later processing (Sidekiq or Que)
@@ -55,6 +62,9 @@ You will also need to install `faraday` or `letter_opener` if you use the `postb
55
62
 
56
63
  ```yaml
57
64
  ---
65
+ :health_check:
66
+ :address: "127.0.0.1"
67
+ :port: 8080
58
68
  :mailboxes:
59
69
  -
60
70
  :email: "user1@gmail.com"
@@ -114,11 +124,103 @@ You will also need to install `faraday` or `letter_opener` if you use the `postb
114
124
  :host: 127.0.0.1
115
125
  :port: 26379
116
126
  :worker: EmailReceiverWorker
127
+ -
128
+ :email: "user7@outlook365.com"
129
+ :password: "password"
130
+ :name: "inbox"
131
+ :inbox_method: microsoft_graph
132
+ :inbox_options:
133
+ :tenant_id: 12345
134
+ :client_id: ABCDE
135
+ :client_secret: YOUR-SECRET-HERE
136
+ :poll_interval: 60
137
+ :delivery_method: sidekiq
138
+ :delivery_options:
139
+ :redis_url: redis://localhost:6379
140
+ :worker: EmailReceiverWorker
117
141
  ```
118
142
 
119
143
  **Note:** If using `delete_after_delivery`, you also probably want to use
120
144
  `expunge_deleted` unless you really know what you're doing.
121
145
 
146
+ ## health_check ##
147
+
148
+ Requires `webrick` gem to be installed.
149
+
150
+ This option enables an HTTP server that listens to a bind address
151
+ defined by `address` and `port`. The following endpoints are supported:
152
+
153
+ * `/liveness`: This returns a 200 status code with `OK` as the body if
154
+ the server is running. Otherwise, it returns a 500 status code.
155
+
156
+ This feature is not included in upstream `mail_room` and is specific to GitLab.
157
+
158
+ ## inbox_method
159
+
160
+ By default, IMAP mode is assumed for reading a mailbox.
161
+
162
+ ### IMAP Server Configuration ##
163
+
164
+ You can set per-mailbox configuration for the IMAP server's `host` (default: 'imap.gmail.com'), `port` (default: 993), `ssl` (default: true), and `start_tls` (default: false).
165
+
166
+ If you want to set additional options for IMAP SSL you can pass a YAML hash to match [SSLContext#set_params](http://docs.ruby-lang.org/en/2.2.0/OpenSSL/SSL/SSLContext.html#method-i-set_params). If you set `verify_mode` to `:none` it'll replace with the appropriate constant.
167
+
168
+ If you're seeing the error `Please log in via your web browser: https://support.google.com/mail/accounts/answer/78754 (Failure)`, you need to configure your Gmail account to allow less secure apps to access it: https://support.google.com/accounts/answer/6010255.
169
+
170
+ ### Microsoft Graph configuration
171
+
172
+ To use the Microsoft Graph API instead of IMAP to read e-mail, you will
173
+ need to create an application in the Azure Active Directory. See the
174
+ [Microsoft instructions](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app) for more details:
175
+
176
+ 1. Sign in to the [Azure portal](https://portal.azure.com).
177
+ 1. Search for and select `Azure Active Directory`.
178
+ 1. Under `Manage`, select `App registrations` > `New registration`.
179
+ 1. Enter a `Name` for your application, such as `MailRoom`. Users of your app might see this name, and you can change it later.
180
+ 1. If `Supported account types` is listed, select the appropriate option.
181
+ 1. Leave `Redirect URI` blank. This is not needed.
182
+ 1. Select `Register`.
183
+ 1. Under `Manage`, select `Certificates & secrets`.
184
+ 1. Under `Client secrets`, select `New client secret`, and enter a name.
185
+ 1. Under `Expires`, select `Never`, unless you plan on updating the credentials every time it expires.
186
+ 1. Select `Add`. Record the secret value in a safe location for use in a later step.
187
+ 1. Under `Manage`, select `API Permissions` > `Add a permission`. Select `Microsoft Graph`.
188
+ 1. Select `Application permissions`.
189
+ 1. Under the `Mail` node, select `Mail.ReadWrite`, and then select Add permissions.
190
+ 1. If `User.Read` is listed in the permission list, you can delete this.
191
+ 1. Click `Grant admin consent` for these permissions.
192
+
193
+ #### Restrict mailbox access
194
+
195
+ Note that for MailRoom to work as a service account, this application
196
+ must have the `Mail.ReadWrite` to read/write mail in *all*
197
+ mailboxes. However, while this appears to be security risk,
198
+ we can configure an application access policy to limit the
199
+ mailbox access for this account. [Follow these instructions](https://docs.microsoft.com/en-us/graph/auth-limit-mailbox-access)
200
+ to setup PowerShell and configure this policy.
201
+
202
+ #### MailRoom config for Microsoft Graph
203
+
204
+ In the MailRoom configuration, set `inbox_method` to `microsoft_graph`.
205
+ You will also need:
206
+
207
+ * The client and tenant ID from the `Overview` section in the Azure app page
208
+ * The client secret created earlier
209
+
210
+ Fill in `inbox_options` with these values:
211
+
212
+ ```yaml
213
+ :inbox_method: microsoft_graph
214
+ :inbox_options:
215
+ :tenant_id: 12345
216
+ :client_id: ABCDE
217
+ :client_secret: YOUR-SECRET-HERE
218
+ :poll_interval: 60
219
+ ```
220
+
221
+ By default, MailRoom will poll for new messages every 60 seconds. `poll_interval` configures the number of
222
+ seconds to poll. Setting the value to 0 or under will default to 60 seconds.
223
+
122
224
  ## delivery_method ##
123
225
 
124
226
  ### postback ###
@@ -263,20 +365,12 @@ it's probably because the content-type is set to Faraday's default, which is `H
263
365
  ## idle_timeout ##
264
366
 
265
367
  By default, the IDLE command will wait for 29 minutes (in order to keep the server connection happy).
266
- If you'd prefer not to wait that long, you can pass `imap_timeout` in seconds for your mailbox configuration.
368
+ If you'd prefer not to wait that long, you can pass `idle_timeout` in seconds for your mailbox configuration.
267
369
 
268
370
  ## Search Command ##
269
371
 
270
372
  This setting allows configuration of the IMAP search command sent to the server. This still defaults 'UNSEEN'. You may find that 'NEW' works better for you.
271
373
 
272
- ## IMAP Server Configuration ##
273
-
274
- You can set per-mailbox configuration for the IMAP server's `host` (default: 'imap.gmail.com'), `port` (default: 993), `ssl` (default: true), and `start_tls` (default: false).
275
-
276
- If you want to set additional options for IMAP SSL you can pass a YAML hash to match [SSLContext#set_params](http://docs.ruby-lang.org/en/2.2.0/OpenSSL/SSL/SSLContext.html#method-i-set_params). If you set `verify_mode` to `:none` it'll replace with the appropriate constant.
277
-
278
- If you're seeing the error `Please log in via your web browser: https://support.google.com/mail/accounts/answer/78754 (Failure)`, you need to configure your Gmail account to allow less secure apps to access it: https://support.google.com/accounts/answer/6010255.
279
-
280
374
  ## Running in Production ##
281
375
 
282
376
  I suggest running with either upstart or init.d. Check out this wiki page for some example scripts for both: https://github.com/tpitale/mail_room/wiki/Init-Scripts-for-Running-mail_room
data/Rakefile CHANGED
@@ -3,4 +3,4 @@ require 'rspec/core/rake_task'
3
3
 
4
4
  RSpec::Core::RakeTask.new(:spec)
5
5
 
6
- task :default => :spec
6
+ task default: :spec
data/lib/mail_room.rb CHANGED
@@ -7,8 +7,10 @@ end
7
7
 
8
8
  require "mail_room/version"
9
9
  require "mail_room/configuration"
10
+ require "mail_room/health_check"
10
11
  require "mail_room/mailbox"
11
12
  require "mail_room/mailbox_watcher"
13
+ require "mail_room/message"
12
14
  require "mail_room/connection"
13
15
  require "mail_room/coordinator"
14
16
  require "mail_room/cli"
@@ -31,7 +31,7 @@ module MailRoom
31
31
  # Any subsequent failure in the instance which gets the lock will be dealt
32
32
  # with by the expiration, at which time another instance can pick up the
33
33
  # message and try again.
34
- client.set(key, 1, {:nx => true, :ex => expiration})
34
+ client.set(key, 1, {nx: true, ex: expiration})
35
35
  end
36
36
 
37
37
  private
data/lib/mail_room/cli.rb CHANGED
@@ -42,7 +42,7 @@ module MailRoom
42
42
  end.parse!(args)
43
43
 
44
44
  self.configuration = Configuration.new(options)
45
- self.coordinator = Coordinator.new(configuration.mailboxes)
45
+ self.coordinator = Coordinator.new(configuration.mailboxes, configuration.health_check)
46
46
  end
47
47
 
48
48
  # Start the coordinator running, sets up signal traps
@@ -4,7 +4,7 @@ module MailRoom
4
4
  # Wraps configuration for a set of individual mailboxes with global config
5
5
  # @author Tony Pitale
6
6
  class Configuration
7
- attr_accessor :mailboxes, :log_path, :quiet
7
+ attr_accessor :mailboxes, :log_path, :quiet, :health_check
8
8
 
9
9
  # Initialize a new configuration of mailboxes
10
10
  def initialize(options={})
@@ -18,6 +18,7 @@ module MailRoom
18
18
  config_file = YAML.load(erb.result)
19
19
 
20
20
  set_mailboxes(config_file[:mailboxes])
21
+ set_health_check(config_file[:health_check])
21
22
  rescue => e
22
23
  raise e unless quiet
23
24
  end
@@ -32,5 +33,14 @@ module MailRoom
32
33
  self.mailboxes << Mailbox.new(attributes)
33
34
  end
34
35
  end
36
+
37
+ # Builds the health checker from YAML configuration
38
+ #
39
+ # @param health_check_config nil or a Hash containing :address and :port
40
+ def set_health_check(health_check_config)
41
+ return unless health_check_config
42
+
43
+ self.health_check = HealthCheck.new(health_check_config)
44
+ end
35
45
  end
36
46
  end
@@ -1,197 +1,21 @@
1
+ # frozen_string_literal: true
2
+
1
3
  module MailRoom
2
4
  class Connection
5
+ attr_reader :mailbox, :new_message_handler
6
+
3
7
  def initialize(mailbox)
4
8
  @mailbox = mailbox
5
-
6
- # log in and set the mailbox
7
- reset
8
- setup
9
9
  end
10
10
 
11
11
  def on_new_message(&block)
12
12
  @new_message_handler = block
13
13
  end
14
14
 
15
- # is the connection logged in?
16
- # @return [Boolean]
17
- def logged_in?
18
- @logged_in
19
- end
20
-
21
- # is the connection blocked idling?
22
- # @return [Boolean]
23
- def idling?
24
- @idling
25
- end
26
-
27
- # is the imap connection closed?
28
- # @return [Boolean]
29
- def disconnected?
30
- imap.disconnected?
31
- end
32
-
33
- # is the connection ready to idle?
34
- # @return [Boolean]
35
- def ready_to_idle?
36
- logged_in? && !idling?
37
- end
38
-
39
- def quit
40
- stop_idling
41
- reset
42
- end
43
-
44
15
  def wait
45
- begin
46
- # in case we missed any between idles
47
- process_mailbox
48
-
49
- idle
50
-
51
- process_mailbox
52
- rescue Net::IMAP::Error, IOError => e
53
- @mailbox.logger.warn({ context: @mailbox.context, action: "Disconnected. Resetting...", error: e.message })
54
- reset
55
- setup
56
- end
57
- end
58
-
59
- private
60
-
61
- def reset
62
- @imap = nil
63
- @logged_in = false
64
- @idling = false
65
- end
66
-
67
- def setup
68
- @mailbox.logger.info({ context: @mailbox.context, action: "Starting TLS session" })
69
- start_tls
70
-
71
- @mailbox.logger.info({ context: @mailbox.context, action: "Logging into mailbox" })
72
- log_in
73
-
74
- @mailbox.logger.info({ context: @mailbox.context, action: "Setting mailbox" })
75
- set_mailbox
76
- end
77
-
78
- # build a net/imap connection to google imap
79
- def imap
80
- @imap ||= Net::IMAP.new(@mailbox.host, :port => @mailbox.port, :ssl => @mailbox.ssl_options)
81
- end
82
-
83
- # start a TLS session
84
- def start_tls
85
- imap.starttls if @mailbox.start_tls
86
- end
87
-
88
- # send the imap login command to google
89
- def log_in
90
- imap.login(@mailbox.email, @mailbox.password)
91
- @logged_in = true
92
- end
93
-
94
- # select the mailbox name we want to use
95
- def set_mailbox
96
- imap.select(@mailbox.name) if logged_in?
97
- end
98
-
99
- # is the response for a new message?
100
- # @param response [Net::IMAP::TaggedResponse] the imap response from idle
101
- # @return [Boolean]
102
- def message_exists?(response)
103
- response.respond_to?(:name) && response.name == 'EXISTS'
104
- end
105
-
106
- # @private
107
- def idle_handler
108
- lambda {|response| imap.idle_done if message_exists?(response)}
16
+ raise NotImplementedError
109
17
  end
110
18
 
111
- # maintain an imap idle connection
112
- def idle
113
- return unless ready_to_idle?
114
-
115
- @mailbox.logger.info({ context: @mailbox.context, action: "Idling" })
116
- @idling = true
117
-
118
- imap.idle(@mailbox.idle_timeout, &idle_handler)
119
- ensure
120
- @idling = false
121
- end
122
-
123
- # trigger the idle to finish and wait for the thread to finish
124
- def stop_idling
125
- return unless idling?
126
-
127
- imap.idle_done
128
-
129
- # idling_thread.join
130
- # self.idling_thread = nil
131
- end
132
-
133
- def process_mailbox
134
- return unless @new_message_handler
135
- @mailbox.logger.info({ context: @mailbox.context, action: "Processing started" })
136
-
137
- msgs = new_messages
138
-
139
- any_deletions = msgs.
140
- # deliver each new message, collect success
141
- map(&@new_message_handler).
142
- # include messages with success
143
- zip(msgs).
144
- # filter failed deliveries, collect message
145
- select(&:first).map(&:last).
146
- # scrub delivered messages
147
- map { |message| scrub(message) }.
148
- any?
149
-
150
- imap.expunge if @mailbox.expunge_deleted && any_deletions
151
- end
152
-
153
- def scrub(message)
154
- if @mailbox.delete_after_delivery
155
- imap.store(message.seqno, "+FLAGS", [Net::IMAP::DELETED])
156
- true
157
- end
158
- end
159
-
160
- # @private
161
- # fetch all messages for the new message ids
162
- def new_messages
163
- # Both of these calls may results in
164
- # imap raising an EOFError, we handle
165
- # this exception in the watcher
166
- messages_for_ids(new_message_ids)
167
- end
168
-
169
- # TODO: label messages?
170
- # @imap.store(id, "+X-GM-LABELS", [label])
171
-
172
- # @private
173
- # search for all new (unseen) message ids
174
- # @return [Array<Integer>] message ids
175
- def new_message_ids
176
- # uid_search still leaves messages UNSEEN
177
- all_unread = imap.uid_search(@mailbox.search_command)
178
-
179
- all_unread = all_unread.slice(0, @mailbox.limit_max_unread) unless @mailbox.limit_max_unread.nil? || @mailbox.limit_max_unread == 0
180
-
181
- to_deliver = all_unread.select { |uid| @mailbox.deliver?(uid) }
182
- @mailbox.logger.info({ context: @mailbox.context, action: "Getting new messages", unread: {count: all_unread.count, ids: all_unread}, to_be_delivered: { count: to_deliver.count, ids: to_deliver } })
183
- to_deliver
184
- end
185
-
186
- # @private
187
- # fetch the email for all given ids in RFC822 format
188
- # @param ids [Array<Integer>] list of message ids
189
- # @return [Array<Net::IMAP::FetchData>] the net/imap messages for the given ids
190
- def messages_for_ids(uids)
191
- return [] if uids.empty?
192
-
193
- # uid_fetch marks as SEEN, will not be re-fetched for UNSEEN
194
- imap.uid_fetch(uids, "RFC822")
195
- end
19
+ def quit; end
196
20
  end
197
21
  end