inbox-sync 0.1.0

data/.gitignore

@@ -0,0 +1,21 @@
+*.gem
+*.log
+*.rbc
+.bundle
+.config
+.yardoc
+.rvmrc
+.rbenv-version
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+test.rb

data/Gemfile

@@ -0,0 +1,7 @@
+source "http://rubygems.org"
+
+# Specify your gem's dependencies in inbox-sync.gemspec
+gemspec
+
+gem 'bundler', '~>1.1'
+gem 'rake',    '~>0.9.2'

data/LICENSE

@@ -0,0 +1,22 @@
+Copyright (c) 2012 Kelly Redding
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,185 @@
+# InboxSync
+
+Move messages from one inbox to another.  Useful when server-side email forwarding is not an option.  (TODO) Can apply filters to messages as they are being moved.  Run on-demand, on a schedule, or as a daemon.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+    gem 'inbox-sync'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install inbox-sync
+
+# How does it work?
+
+InboxSync uses IMAP to query a source inbox, process its messages, append them to a destination inbox, and archive them on the source.  It logs each step in the process and will send notification emails when something goes wrong.
+
+(TODO) InboxSync provides a framework for defining destination filters for post-sync mail processing (ie moving/archiving, copying/labeling, deletion, etc).
+
+InboxSync provides a basic ruby runner class to handle polling the source on an interval and running the configured sync(s).  You can call it in any number of ways: in a script, from a cron, as a daemon, or as part of a larger system.
+
+## Usage
+
+It should be fairly straight-forward: create and configure a sync then run it.  This will move all messages in the `source` inbox to the `dest` inbox.
+
+### Create your Sync
+
+```ruby
+sync = InboxSync.new
+```
+
+### Configure it
+
+```ruby
+# manually set configs
+sync.config.source.host = 'imap.source-host.com'
+
+# or use a more DSL like approach
+sync.config.source.login.user 'me'
+sync.config.source.login.pw   'secret'
+
+# or use a configure block, if you like
+sync.configure do
+  dest.host  'imap.dest-host.com'
+  dest.login 'me', 'secret'
+end
+```
+
+### Run it
+
+```ruby
+InboxSync.run(sync, :interval => 5)
+```
+
+## Sync Definition
+
+### `source`
+
+IMAP settings for the source inbox.
+
+* *host*: eg. `'imap.some-domain.com'`.
+* *port*: defaults to `143`.
+* *ssl*:  whether to use SSL.  defaults to `false`.
+* *login*: credentials (user, pw).
+* *inbox*: name of the inbox folder.  defaults to `'INBOX'`
+* *expunge*: whether to expunge the inbox before and after processing.  defaults to `true`.
+
+### `dest`
+
+IMAP settings for the destination inbox.  Has the some attributes and defaults as the `source`.
+
+### `notify`
+
+SMTP settings to send notifications with.
+
+* *host*: eg. `'smtp.some-domain.com'`.
+* *port*: defaults to `25`.
+* *tls*: whethe to use TLS encryption.  defaults to `false`.
+* *helo*: the helo domain to send with.
+* *login*: credentials (user, pw).
+* *authtype*: defaults to `:login`.
+* *from_addr*: address to send the notifications from.
+* *to_addr*: address(es) to send the notifications to.
+
+### `archive_folder`
+
+The (optional) folder on the source to create and archive (move) source inbox messages to when processing is complete.  Defaults to `"Archived"`.  Set to `nil` to disable archiving on the source and delete the messages after processing.
+
+### `logger`
+
+A logger to use.  Defaults to ruby's `Logger` on `STDOUT`.
+
+## Running
+
+InboxSync provides a `Runner` class that will loop indefinitely, running syncs every `:interval` seconds.  Stick it in a daemon, a rake task, a CLI, or whatever depending on how you want to invoke it.  Here is an example using it in a basic ruby script:
+
+```ruby
+require 'inbox-sync'
+
+sync = InboxSync.new.configure do
+  source.host  'imap.gmail.com'
+  source.port  993
+  source.ssl   'Yes'
+  source.login 'joetest@kellyredding.com', 'joetest1'
+
+  dest.host  'imap.gmail.com'
+  dest.port  993
+  dest.ssl   'Yes'
+  dest.login 'suetest@kellyredding.com', 'suetest1'
+
+  notify.host    'smtp.gmail.com'
+  notify.port    587
+  notify.tls     'Yes'
+  notify.helo    'gmail.com'
+  notify.login   'joetest@kellyredding.com', 'joetest1'
+  notify.to_addr 'joetest@kellyredding.com'
+  notify.to_addr 'suetest@kellyredding.com'
+
+  logger Logger.new('log/inbox-sync.log')
+end
+
+InboxSync.run(sync, :interval => 20)
+```
+
+The `InboxSync.run` method is just a macro for creating a runner and calling its `start` method.
+
+```ruby
+InboxSync::Runner.new(sync, :interval => 5).start
+```
+
+By default, it will log to `STDOUT` but accepts a `:logger` option to override this.
+
+```ruby
+InboxSync.run(sync, {
+  :interval => 5,
+  :logger => Logger.new('/path/to/log.log')
+})
+```
+
+You can pass any number of syncs to run.  Each `:interval` period, it will run them sequentially:
+
+```ruby
+InboxSync.run(sync1, sync2, sync3, :interval => 5)
+```
+
+If you pass no `:interval` option (or pass a negative value for it), the runner will run the sync(s) once and then exit instead of running the syncs indefinitely on the interval.
+
+```ruby
+InboxSync.run(sync)
+```
+
+The runner traps `SIGINT` and `SIGQUIT` and will shutdown nicely once any in-progress syncs have finished.
+
+## Filter Framework
+
+TODO
+
+## Error Handling
+
+InboxSync generates detailed logs of both running its syncs and processing sync mail items.  If a mail fails to append (ie rejected by the dest IMAP), InboxSync will attempt to strip the mail to its most basic (ie plain/text) form and will retry the append.
+
+In addtion, InboxSync will notify via email when something goes wrong with a sync.  You configure `notify` settings when defining your syncs.  These settings determine where/how notifications are sent out.  There are two types a notifications InboxSync will send: `RunSyncError` and `SyncMailItemError`.
+
+In any case, if an `archive_folder` is set, no source messages will be permanently deleted and are always available there for reference.
+
+### `RunSyncError` notification
+
+This notification is sent when there is a problem running a sync in general.  For example, the sync can't connect to the source to read its mail items or the runner itself has a runtime exception.  This notification lets you know that something went wrong and that mail items aren't being sync'd.  It also details the exception that happened with a full backtrace.
+
+### `SyncMailItemError` notification
+
+This notification is sent wnen there is a problem syncing a specific mail item.  For example the destination rejects the append or there was a problem archiving the mail item at the source.  It lets you know there was a problem and gives you some info about the email that had a problem.  It also details the exception that happened with a full backtrace.
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Added some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/Rakefile

@@ -0,0 +1,8 @@
+#!/usr/bin/env rake
+
+require 'assert/rake_tasks'
+Assert::RakeTasks.for(:test)
+
+require 'bundler/gem_tasks'
+
+task :default => :build

data/inbox-sync.gemspec

@@ -0,0 +1,23 @@
+# -*- encoding: utf-8 -*-
+require File.expand_path('../lib/inbox-sync/version', __FILE__)
+
+Gem::Specification.new do |gem|
+  gem.name        = "inbox-sync"
+  gem.version     = InboxSync::VERSION
+  gem.description = %q{Move messages from one inbox to another}
+  gem.summary     = %q{Move messages from one inbox to another}
+
+  gem.authors     = ["Kelly Redding"]
+  gem.email       = ["kelly@kellyredding.com"]
+  gem.homepage    = "http://github.com/kellyredding/inbox-sync"
+
+  gem.files         = `git ls-files`.split("\n")
+  gem.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
+  gem.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
+  gem.require_paths = ["lib"]
+
+  gem.add_development_dependency("assert")
+
+  gem.add_dependency("ns-options", ["~> 0.4.1"])
+  gem.add_dependency("mail", ["~> 2.4"])
+end

data/lib/inbox-sync/config/credentials.rb

@@ -0,0 +1,29 @@
+require 'ns-options'
+
+module InboxSync; end
+class InboxSync::Config
+
+  class Credentials
+    include NsOptions::Proxy
+
+    opt :user, :required => true
+    opt :pw, :required => true
+
+    def initialize(*args)
+      the_args = args.flatten
+      if the_args.size == 1
+        self.apply(args.last)
+      else
+        self.user, self.pw = the_args
+      end
+    end
+
+    def validate!
+      if !required_set?
+        raise ArgumentError, "some required configs are missing"
+      end
+    end
+
+  end
+
+end

data/lib/inbox-sync/config/imap_config.rb

@@ -0,0 +1,28 @@
+require 'ns-options'
+require 'ns-options/boolean'
+require 'inbox-sync/config/credentials'
+
+module InboxSync; end
+class InboxSync::Config
+
+  class IMAPConfig
+    include NsOptions::Proxy
+
+    opt :host, :required => true
+    opt :port, :default => 143, :required => true
+    opt :ssl, NsOptions::Boolean, :default => false, :required => true
+    opt :login, Credentials, :required => true, :default => {}
+    opt :inbox, :default => "INBOX", :required => true
+    opt :expunge, NsOptions::Boolean, :default => true, :required => true
+
+    def validate!
+      if !required_set?
+        raise ArgumentError, "some required configs are missing"
+      end
+
+      login.validate!
+    end
+
+  end
+
+end

data/lib/inbox-sync/config/smtp_config.rb

@@ -0,0 +1,30 @@
+require 'ns-options'
+require 'ns-options/boolean'
+require 'inbox-sync/config/credentials'
+
+module InboxSync; end
+class InboxSync::Config
+
+  class SMTPConfig
+    include NsOptions::Proxy
+
+    opt :host, :required => true
+    opt :port, :default => 25, :required => true
+    opt :tls,  NsOptions::Boolean, :default => false, :required => true
+    opt :helo, :required => true
+    opt :login, Credentials, :required => true, :default => {}
+    opt :authtype, :default => :login, :required => true
+    opt :from_addr, :required => true
+    opt :to_addr, :required => true
+
+    def validate!
+      if !required_set?
+        raise ArgumentError, "some required configs are missing"
+      end
+
+      login.validate!
+    end
+
+  end
+
+end

data/lib/inbox-sync/config.rb

@@ -0,0 +1,30 @@
+require 'logger'
+require 'ns-options'
+require 'inbox-sync/config/imap_config'
+require 'inbox-sync/config/smtp_config'
+
+module InboxSync
+
+  class Config
+    include NsOptions::Proxy
+
+    opt :source, IMAPConfig, :required => true, :default => {}
+    opt :dest,   IMAPConfig, :required => true, :default => {}
+    opt :notify, SMTPConfig, :required => true, :default => {}
+
+    opt :archive_folder, :default => 'Archived'
+    opt :logger, Logger, :required => true, :default => STDOUT
+
+    def validate!
+      if !required_set?
+        raise ArgumentError, "some required configs are missing"
+      end
+
+      source.validate!
+      dest.validate!
+      notify.validate!
+    end
+
+  end
+
+end

data/lib/inbox-sync/mail_item.rb

@@ -0,0 +1,78 @@
+require 'mail'
+
+module InboxSync
+
+  class MailItem
+
+    def self.find(imap)
+      imap.uid_search(['ALL']).
+        map do |uid|
+          [uid, imap.uid_fetch(uid, ['RFC822', 'INTERNALDATE']).first]
+        end.
+        map do |uid_meta|
+          self.new(
+            uid_meta.first,
+            uid_meta.last.attr['RFC822'],
+            uid_meta.last.attr["INTERNALDATE"]
+          )
+        end
+    end
+
+    attr_reader :uid, :meta, :message
+
+    def initialize(uid, rfc822, internal_date)
+      @uid = uid
+      @meta = {
+        'RFC822' => rfc822,
+        'INTERNALDATE' => internal_date
+      }
+      @message = ::Mail.new(rfc822)
+    end
+
+    def name
+      "[#{@uid}] #{@message.from}: #{@message.subject.inspect} (#{time_s(@message.date)})"
+    end
+
+    # Returns a stripped down version of the mail item
+    # The stripped down versions is just the 'text/plain' part of multipart
+    # mail items.  If the original mail item was not multipart, then the
+    # stripped down version is the same as the original.
+    # This implies that stripped down mail items have no attachments.
+
+    def stripped
+      @stripped ||= strip_down(MailItem.new(
+        self.uid,
+        self.meta['RFC822'],
+        self.meta["INTERNALDATE"]
+      ))
+    end
+
+    def inspect
+      "#<#{self.class}:#{'0x%x' % (self.object_id << 1)}: @uid=#{@uid.inspect}, from=#{@message.from.inspect}, subject=#{@message.subject.inspect}, 'INTERNALDATE'=#{@meta['INTERNALDATE'].inspect}>"
+    end
+
+    private
+
+    def time_s(datetime)
+      datetime.strftime("%a %b %-d %Y, %I:%M %p")
+    end
+
+    def strip_down(mail_item)
+      message = mail_item.message
+      if message.multipart?
+        message.parts.delete_if do |part|
+          !part.content_type.match(/text\/plain/)
+        end
+        message.parts.first.body = strip_down_body_s(message.parts.first.body)
+        mail_item.meta['RFC822'] = message.to_s
+      end
+      mail_item
+    end
+
+    def strip_down_body_s(body_s)
+      "**[inbox-sync] stripped down to just plain text part**\n\n#{body_s}"
+    end
+
+  end
+
+end

data/lib/inbox-sync/notice/base.rb

@@ -0,0 +1,47 @@
+require 'mail'
+
+module InboxSync; end
+module InboxSync::Notice
+
+  class Base
+
+    attr_reader :mail
+
+    def initialize(smtp, config)
+      @smtp = smtp
+      @config = config
+
+      @mail = ::Mail.new
+      @mail.from = self.from
+      @mail.to = self.to
+      @mail.subject = self.subject
+      @mail.body = self.body
+    end
+
+    def from; @config.from_addr; end
+    def to;   @config.to_addr;   end
+
+    def subject(msg="notice")
+      "[inbox-sync] #{msg}"
+    end
+
+    def body
+      raise RuntimeError, "subclass `Notice::Base` and define your body"
+    end
+
+    def send
+      @smtp.start(helo, user, pw, authtype) do |smtp|
+        smtp.send_message(@mail.to_s, from, to)
+      end
+    end
+
+    protected
+
+    def helo;     @config.helo;       end
+    def user;     @config.login.user; end
+    def pw;       @config.login.pw;   end
+    def authtype; @config.authtype;   end
+
+  end
+
+end

data/lib/inbox-sync/notice/run_sync_error.rb

@@ -0,0 +1,44 @@
+require 'inbox-sync/notice/base'
+
+module InboxSync; end
+module InboxSync::Notice
+
+  class RunSyncError < Base
+
+    BODY = %{
+:sync_name
+
+An error happened while running this sync.  The error has
+been logged but no mail items from this sync's source are
+being sync'd.  The runner will continue to attempt this
+sync so mails like this will continue until the problem
+is fixed.
+
+Error
+=====
+  :error_message (:error_name)
+  :error_backtrace
+    }.strip.freeze
+
+    def initialize(smtp, config, data={})
+      @error = data[:error]
+      @sync = data[:sync]
+
+      super(smtp, config)
+    end
+
+    def subject
+      super("sync run error (#{@sync.uid})")
+    end
+
+    def body
+      @body ||= BODY.
+        gsub(':sync_name', @sync.name).
+        gsub(':error_message', @error.message).
+        gsub(':error_name', @error.class.name).
+        gsub(':error_backtrace', @error.backtrace.join("\n  "))
+    end
+
+  end
+
+end

data/lib/inbox-sync/notice/sync_mail_item_error.rb

@@ -0,0 +1,45 @@
+require 'inbox-sync/notice/base'
+
+module InboxSync; end
+module InboxSync::Notice
+
+  class SyncMailItemError < Base
+
+    BODY = %{
+:sync_name
+:mail_item_name
+
+An error happened while syncing this mail item.  The error
+has been logged and the mail item has been archived on the
+source.  The sync will continue processing new mail items.
+
+Error
+=====
+  :error_message (:error_name)
+  :error_backtrace
+    }.strip.freeze
+
+    def initialize(smtp, config, data={})
+      @error = data[:error]
+      @mail_item = data[:mail_item]
+      @sync = data[:sync]
+
+      super(smtp, config)
+    end
+
+    def subject
+      super("mail item sync error (#{@mail_item.uid}, #{@sync.uid})")
+    end
+
+    def body
+      @body ||= BODY.
+        gsub(':sync_name', @sync.name).
+        gsub(':mail_item_name', @mail_item.name).
+        gsub(':error_message', @error.message).
+        gsub(':error_name', @error.class.name).
+        gsub(':error_backtrace', @error.backtrace.join("\n  "))
+    end
+
+  end
+
+end