gitlab-mail_room 0.0.2

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 83ebaaed437f93cd2df69d48f6542842364fdab407bbf6620347b7a9e0d95199
+  data.tar.gz: d374aee24abf12a1cd274c3dbc9bd9e9feeb440b0eec314807487163617e9cab
+SHA512:
+  metadata.gz: c76aa09decd47a81ee95f12d9a9c0470f48caabaf1443e461784677c58f8685a394198618498ae61d663819af4a99be4a359d5397e05f6cbab456973b995a3d8
+  data.tar.gz: f9815fff30eea223d91046f15e84d2fd4f584114c3a18743ba9f54134aa831138bb8f0936747130b5bc8da8f41b30d6e953a14c8ad4b3bc997bdb62d3e5b447e

data/.gitignore

@@ -0,0 +1,18 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+log

data/.gitlab-ci.yml

@@ -0,0 +1,27 @@
+# Cache gems in between builds
+
+.test-template: &test
+  cache:
+    paths:
+      - vendor/ruby
+  script:
+  - bundle exec rspec spec
+  before_script:
+    - apt update && apt install -y libicu-dev
+    - ruby -v                                   # Print out ruby version for debugging
+    # Uncomment next line if your rails app needs a JS runtime:
+    # - apt-get update -q && apt-get install nodejs -yqq
+    - gem install bundler  --no-document    # Bundler is not installed with the image
+    - bundle install -j $(nproc) --path vendor  # Install dependencies into ./vendor/ruby
+
+rspec-2.4:
+  image: "ruby:2.4"
+  <<: *test
+
+rspec-2.5:
+  image: "ruby:2.5"
+  <<: *test
+
+rspec-2.6:
+  image: "ruby:2.6"
+  <<: *test

data/.ruby-version

@@ -0,0 +1 @@
+2.6.0

data/.travis.yml

@@ -0,0 +1,10 @@
+language: ruby
+rvm:
+  - 2.3.0
+  - 2.4
+  - 2.5
+  - 2.6
+services:
+  - redis-server
+script: bundle exec rspec spec
+sudo: false

data/CHANGELOG.md

@@ -0,0 +1,125 @@
+## mail_room 0.10.0 ##
+
+* Remove imap backports
+* Increase minimum ruby version to 2.3
+* Postback basic_auth support - PR#92
+* Docs for ActionMailbox - PR#92
+* Configuration option for delivery_klass - PR#93
+* Expunge deleted - PR#90
+* Raise error on a few fields of missing configuration - PR#89
+* Remove fakeredis gem - PR#87
+
+    *Tony Pitale <@tpitale>*
+
+* Fix redis arbitration to use NX+EX - PR#86
+
+    *Craig Miskell <@craigmiskell-gitlab>*
+
+* Structured (JSON) logger - PR#88
+
+    *charlie <@cablett>*
+
+## mail_room 0.9.1 ##
+
+* __FILE__ support in yml ERb config - PR#80
+
+    *Gabriel Mazetto <@brodock>*
+
+## mail_room 0.9.0 ##
+
+* Redis Sentinel configuration support - PR#79
+
+    *Gabriel Mazetto <@brodock>*
+
+## mail_room 0.8.1 ##
+
+* Check watching thread exists before joining - PR#78
+
+    *Michal Galet <@galet>*
+
+## mail_room 0.8.0 ##
+
+* Rework the mailbox watcher and handler into a new Connection class to abstract away IMAP handling details
+
+    *Tony Pitale <@tpitale>*
+
+## mail_room 0.7.0 ##
+
+* Backports idle timeout from ruby 2.3.0
+* Sets default to 29 minutes to prevent IMAP disconnects
+* Validates that the timeout does not exceed 29 minutes
+
+    *Tony Pitale <@tpitale>*
+
+## mail_room 0.6.1 ##
+
+* ERB parsing of configuration yml file to enable using ENV variables
+
+    *Douwe Maan <@DouweM>*
+
+## mail_room 0.6.0 ##
+
+* Add redis Arbitration to reduce multiple deliveries of the same message when running multiple MailRoom instances on the same inbox
+
+    *Douwe Maan <@DouweM>*
+
+## mail_room 0.5.2 ##
+
+* Fix Sidekiq delivery method for non-UTF8 email
+
+    *Douwe Maan <@DouweM>*
+
+* Add StartTLS session support
+
+    *Tony Pitale <@tpitale>*
+
+## mail_room 0.5.1 ##
+
+* Re-idle after 29 minutes to maintain IDLE connection
+
+    *Douwe Maan <@DouweM>*
+
+## mail_room 0.5.0 ##
+
+* Que delivery method
+
+    *Tony Pitale <@tpitale>*
+
+## mail_room 0.4.2 ##
+
+* rescue from all IOErrors, not just EOFError
+
+    *Douwe Maan <@DouweM>*
+
+## mail_room 0.4.1 ##
+
+* Fix redis default host/port configuration
+* Mailbox does not attempt delivery without a message
+
+    *Douwe Maan <@DouweM>*
+
+## mail_room 0.4.0 ##
+
+* Sidekiq delivery method
+* Option to delete messages after delivered
+
+    *Douwe Maan <@DouweM>*
+
+* -q/--quiet do not raise errors on missing configuration
+* prefetch mail messages before idling
+* delivery-method-specific delivery options configuration
+
+    *Tony Pitale <@tpitale>*
+
+## mail_room 0.3.1 ##
+
+* Rescue from EOFError and re-setup mailroom
+
+    *Tony Pitale <@tpitale>*
+
+## mail_room 0.3.0 ##
+
+*   Reconnect and idle if disconnected during an existing idle.
+*   Set idling thread to abort on exception so any unhandled exceptions will stop mail_room running.
+
+    *Tony Pitale <@tpitale>*

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,24 @@
+# Contributor Code of Conduct
+
+As contributors and maintainers of this project, and in the interest of fostering an open and welcoming community, we pledge to respect all people who contribute through reporting issues, posting feature requests, updating documentation, submitting pull requests or patches, and other activities.
+
+We are committed to making participation in this project a harassment-free experience for everyone, regardless of level of experience, gender, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, religion, or nationality.
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery
+* Personal attacks
+* Trolling or insulting/derogatory comments
+* Public or private harassment
+* Publishing other's private information, such as physical or electronic addresses, without explicit permission
+* Other unethical or unprofessional conduct
+
+Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
+
+By adopting this Code of Conduct, project maintainers commit themselves to fairly and consistently applying these principles to every aspect of managing this project. Project maintainers who do not follow or enforce the Code of Conduct may be permanently removed from the project team.
+
+This code of conduct applies both within project spaces and in public spaces when an individual is representing the project or its community.
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting a project maintainer at [INSERT EMAIL ADDRESS]. All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. Maintainers are obligated to maintain confidentiality with regard to the reporter of an incident.
+
+This Code of Conduct is adapted from the [Contributor Covenant](http://contributor-covenant.org), version 1.3.0, available at [http://contributor-covenant.org/version/1/3/0/](http://contributor-covenant.org/version/1/3/0/)

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in mail_room.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2013 Tony Pitale
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,361 @@
+# mail_room #
+
+## Fork notice
+
+mail_room contains some merged functionality that GitLab requires, so this mirror fork is to help us release custom functionality.
+
+## README
+
+mail_room is a configuration based process that will idle on IMAP connections and execute a delivery method when a new message is received. Examples of delivery methods include:
+
+* POST to a delivery URL (Postback)
+* Queue a job to Sidekiq or Que for later processing (Sidekiq or Que)
+* Log the message or open with LetterOpener (Logger or LetterOpener)
+
+[![Build Status](https://travis-ci.org/tpitale/mail_room.png?branch=master)](https://travis-ci.org/tpitale/mail_room)
+[![Code Climate](https://codeclimate.com/github/tpitale/mail_room/badges/gpa.svg)](https://codeclimate.com/github/tpitale/mail_room)
+
+## Installation ##
+
+Add this line to your application's Gemfile:
+
+    gem 'mail_room'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install mail_room
+
+You will also need to install `faraday` or `letter_opener` if you use the `postback` or `letter_opener` delivery methods, respectively.
+
+## Usage ##
+
+    mail_room -c /path/to/config.yml
+
+**Note:** To ignore missing config file or missing `mailboxes` key, use `-q` or `--quiet`
+
+## Configuration ##
+
+```yaml
+---
+:mailboxes:
+  -
+    :email: "user1@gmail.com"
+    :password: "password"
+    :name: "inbox"
+    :search_command: 'NEW'
+    :logger:
+      :log_path: /path/to/logfile/for/mailroom
+    :delivery_options:
+      :delivery_url: "http://localhost:3000/inbox"
+      :delivery_token: "abcdefg"
+      :content_type: "text/plain"
+
+  -
+    :email: "user2@gmail.com"
+    :password: "password"
+    :name: "inbox"
+    :delivery_method: postback
+    :delivery_options:
+      :delivery_url: "http://localhost:3000/inbox"
+      :delivery_token: "abcdefg"
+  -
+    :email: "user3@gmail.com"
+    :password: "password"
+    :name: "inbox"
+    :delivery_method: logger
+    :delivery_options:
+      :log_path: "/var/log/user3-email.log"
+  -
+    :email: "user4@gmail.com"
+    :password: "password"
+    :name: "inbox"
+    :delivery_method: letter_opener
+    :delete_after_delivery: true
+    :expunge_deleted: true
+    :delivery_options:
+      :location: "/tmp/user4-email"
+  -
+    :email: "user5@gmail.com"
+    :password: "password"
+    :name: "inbox"
+    :delivery_method: sidekiq
+    :delivery_options:
+      :redis_url: redis://localhost:6379
+      :worker: EmailReceiverWorker
+  -
+    :email: "user6@gmail.com"
+    :password: "password"
+    :name: "inbox"
+    :delivery_method: sidekiq
+    :delivery_options:
+      # When pointing to sentinel, follow this sintax for redis URLs:
+      # redis://:<password>@<master-name>/
+      :redis_url: redis://:password@my-redis-sentinel/
+      :sentinels:
+        -
+          :host: 127.0.0.1
+          :port: 26379
+      :worker: EmailReceiverWorker
+```
+
+**Note:** If using `delete_after_delivery`, you also probably want to use
+`expunge_deleted` unless you really know what you're doing.
+
+## delivery_method ##
+
+### postback ###
+
+Requires `faraday` gem be installed.
+
+*NOTE:* If you're using Ruby `>= 2.0`, you'll need to use Faraday from `>= 0.8.9`. Versions before this seem to have some weird behavior with `mail_room`.
+
+The default delivery method, requires `delivery_url` and `delivery_token` in
+configuration.
+
+You can pass `content_type:` option to overwrite `faraday's` default content-type(`application/x-www-form-urlencoded`) for post requests, we recommend passing `text/plain` as content-type.
+
+As the postback is essentially using your app as if it were an API endpoint,
+you may need to disable forgery protection as you would with a JSON API.
+
+### sidekiq ###
+
+Deliver the message by pushing it onto the configured Sidekiq queue to be handled by a custom worker.
+
+Requires `redis` gem to be installed.
+
+Configured with `:delivery_method: sidekiq`.
+
+Delivery options:
+- **redis_url**: The Redis server to connect with. Use the same Redis URL that's used to configure Sidekiq.
+  Required, defaults to `redis://localhost:6379`.
+- **sentinels**: A list of sentinels servers used to provide HA to Redis. (see [Sentinel Support](#sentinel-support))
+  Optional.
+- **namespace**: The Redis namespace Sidekiq works under. Use the same Redis namespace that's used to configure Sidekiq.
+  Optional.
+- **queue**: The Sidekiq queue the job is pushed onto. Make sure Sidekiq actually reads off this queue.
+  Required, defaults to `default`.
+- **worker**: The worker class that will handle the message.
+  Required.
+
+An example worker implementation looks like this:
+
+
+```ruby
+class EmailReceiverWorker
+  include Sidekiq::Worker
+
+  def perform(message)
+    mail = Mail::Message.new(message)
+
+    puts "New mail from #{mail.from.first}: #{mail.subject}"
+  end
+end
+```
+
+### que ###
+
+Deliver the message by pushing it onto the configured Que queue to be handled by a custom worker.
+
+Requires `pg` gem to be installed.
+
+Configured with `:delivery_method: que`.
+
+Delivery options:
+- **host**: The postgresql server host to connect with. Use the database you use with Que.
+  Required, defaults to `localhost`.
+- **port**: The postgresql server port to connect with. Use the database you use with Que.
+  Required, defaults to `5432`.
+- **database**: The postgresql database to use. Use the database you use with Que.
+  Required.
+- **queue**: The Que queue the job is pushed onto. Make sure Que actually reads off this queue.
+  Required, defaults to `default`.
+- **job_class**: The worker class that will handle the message.
+  Required.
+- **priority**: The priority you want this job to run at.
+  Required, defaults to `100`, lowest Que default priority.
+
+An example worker implementation looks like this:
+
+```ruby
+class EmailReceiverJob < Que::Job
+  def run(message)
+    mail = Mail::Message.new(message)
+
+    puts "New mail from #{mail.from.first}: #{mail.subject}"
+  end
+end
+```
+
+### logger ###
+
+Configured with `:delivery_method: logger`.
+
+If the `:log_path:` delivery option is not provided, defaults to `STDOUT`
+
+### noop ###
+
+Configured with `:delivery_method: noop`.
+
+Does nothing, like it says.
+
+### letter_opener ###
+
+Requires `letter_opener` gem be installed.
+
+Configured with `:delivery_method: letter_opener`.
+
+Uses Ryan Bates' excellent [letter_opener](https://github.com/ryanb/letter_opener) gem.
+
+## ActionMailbox in Rails ##
+
+MailRoom can deliver mail to Rails using the ActionMailbox [configuration options for an SMTP relay](https://edgeguides.rubyonrails.org/action_mailbox_basics.html#configuration).
+
+In summary (from the ActionMailbox docs)
+
+1. Configure Rails to use the `:relay` ingress option:
+```rb
+# config/environments/production.rb
+config.action_mailbox.ingress = :relay
+```
+
+2. Generate a strong password (e.g., using SecureRandom or something) and add it to Rails config:
+using `rails credentials:edit` under `action_mailbox.ingress_password`.
+
+And finally, configure MailRoom to use the postback configuration with the options:
+
+```yaml
+:delivery_method: postback
+:delivery_options:
+  :delivery_url: https://example.com/rails/action_mailbox/relay/inbound_emails
+  :delivery_username: actionmailbox
+  :delivery_password: <INGRESS_PASSWORD>
+```
+
+## Receiving `postback` in Rails ##
+
+If you have a controller that you're sending to, with forgery protection
+disabled, you can get the raw string of the email using `request.body.read`.
+
+I would recommend having the `mail` gem bundled and parse the email using
+`Mail.read_from_string(request.body.read)`.
+
+*Note:* If you get the exception (`Rack::QueryParser::InvalidParameterError (invalid %-encoding...`)
+it's probably because the content-type is set to Faraday's default, which is  `HEADERS['content-type'] = 'application/x-www-form-urlencoded'`. It can cause `Rack` to crash due to `InvalidParameterError` exception. When you send a post with `application/x-www-form-urlencoded`, `Rack` will attempt to parse the input and can end up raising an exception, for example if the email that you are forwarding contain `%%` in its content or headers it will cause Rack to crash with the message above.
+
+## idle_timeout ##
+
+By default, the IDLE command will wait for 29 minutes (in order to keep the server connection happy).
+If you'd prefer not to wait that long, you can pass `imap_timeout` in seconds for your mailbox configuration.
+
+## Search Command ##
+
+This setting allows configuration of the IMAP search command sent to the server. This still defaults 'UNSEEN'. You may find that 'NEW' works better for you.
+
+## IMAP Server Configuration ##
+
+You can set per-mailbox configuration for the IMAP server's `host` (default: 'imap.gmail.com'), `port` (default: 993), `ssl` (default: true), and `start_tls` (default: false).
+
+If you want to set additional options for IMAP SSL you can pass a YAML hash to match [SSLContext#set_params](http://docs.ruby-lang.org/en/2.2.0/OpenSSL/SSL/SSLContext.html#method-i-set_params). If you set `verify_mode` to `:none` it'll replace with the appropriate constant.
+
+If you're seeing the error `Please log in via your web browser: https://support.google.com/mail/accounts/answer/78754 (Failure)`, you need to configure your Gmail account to allow less secure apps to access it: https://support.google.com/accounts/answer/6010255.
+
+## Running in Production ##
+
+I suggest running with either upstart or init.d. Check out this wiki page for some example scripts for both: https://github.com/tpitale/mail_room/wiki/Init-Scripts-for-Running-mail_room
+
+## Arbitration ##
+
+When running multiple instances of MailRoom against a single mailbox, to try to prevent delivery of the same message multiple times, we can configure Arbitration using Redis.
+
+```yaml
+:mailboxes:
+  -
+    :email: "user1@gmail.com"
+    :password: "password"
+    :name: "inbox"
+    :delivery_method: postback
+    :delivery_options:
+      :delivery_url: "http://localhost:3000/inbox"
+      :delivery_token: "abcdefg"
+
+    :arbitration_method: redis
+    :arbitration_options:
+      # The Redis server to connect with. Defaults to redis://localhost:6379.
+      :redis_url: redis://redis.example.com:6379
+      # The Redis namespace to house the Redis keys under. Optional.
+      :namespace: mail_room
+  -
+    :email: "user2@gmail.com"
+    :password: "password"
+    :name: "inbox"
+    :delivery_method: postback
+    :delivery_options:
+      :delivery_url: "http://localhost:3000/inbox"
+      :delivery_token: "abcdefg"
+
+    :arbitration_method: redis
+    :arbitration_options:
+      # When pointing to sentinel, follow this sintax for redis URLs:
+      # redis://:<password>@<master-name>/
+      :redis_url: redis://:password@my-redis-sentinel/
+      :sentinels:
+        -
+          :host: 127.0.0.1
+          :port: 26379
+      # The Redis namespace to house the Redis keys under. Optional.
+      :namespace: mail_room
+```
+
+**Note:** This will likely never be a _perfect_ system for preventing multiple deliveries of the same message, so I would advise checking the unique `message_id` if you are running in this situation.
+
+**Note:** There are other scenarios for preventing duplication of messages at scale that _may_ be more appropriate in your particular setup. One such example is using multiple inboxes in reply-by-email situations. Another is to use labels and configure a different `SEARCH` command for each instance of MailRoom.
+
+## Sentinel Support
+
+Redis Sentinel provides high availability for Redis. Please read their [documentation](http://redis.io/topics/sentinel)
+first, before enabling it with mail_room.
+
+To connect to a Sentinel, you need to setup authentication to both sentinels and redis daemons first, and make sure
+both are binding to a reachable IP address.
+
+In mail_room, when you are connecting to a Sentinel, you have to inform the `master-name` and the `password` through
+`redis_url` param, following this syntax:
+
+```
+redis://:<password>@<master-name>/
+```
+
+You also have to inform at least one pair of `host` and `port` for a sentinel in your cluster.
+To have a minimum reliable setup, you need at least `3` sentinel nodes and `3` redis servers (1 master, 2 slaves).
+
+## Logging ##
+
+MailRoom will output JSON-formatted logs to give some observability into its operations.
+
+Simply configure a `log_path` for the `logger` on any of your mailboxes. By default, nothing will be logged.
+
+If you wish to log to `STDOUT` or `STDERR` instead of a file, you can pass `:stdout` or `:stderr`,
+respectively and MailRoom will log there.
+
+## Contributing ##
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request
+6. If accepted, ask for commit rights
+
+## TODO ##
+
+1. specs, this is just a (working) proof of concept √
+2. finish code for POSTing to callback with auth √
+3. accept mailbox configuration for one account directly on the commandline; or ask for it
+4. add example rails endpoint, with auth examples
+5. add example configs for upstart/init.d √
+6. log to stdout √
+7. add a development mode that opens in letter_opener by ryanb √