winton-fetcher 0.1.1

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2007 Dan Weinand and Slantwise Design, LLC
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.rdoc

@@ -0,0 +1,94 @@
+= Fetcher
+
+Fetcher is a simple message fetcher perfect for using in a daemon or via cron.
+
+It implements the following common pattern:
+
+1. Connect to a server
+2. Download available messages
+3. Send each message to another object for further processing
+4. Remove downloaded messages from the remote server
+
+Install using:
+
+  sudo gem install winton-fetcher
+
+== Notice
+
+This is a fork of {fetcher}[http://github.com/look/fetcher] by {Luke Francl}[http://github.com/look].
+
+Differences:
+
+* Gemified
+* Doesn't have generators
+* Fixes a connection exception issue
+
+== Usage
+
+Create a new fetcher object like the following:
+
+  @fetcher = Fetcher.create({:type => :pop,
+                             :receiver => IncomingMailHandler,
+                             :server => 'mail.example.com',
+                             :username => 'jim',
+                             :password => 'test'})
+
+The receiver object is expected to have a receive method that takes a message as its only argument 
+(e.g., the way <tt>ActionMailer::Base.recieve</tt> works; but you don't <em>have</em> to use ActionMailer.).
+
+Call <tt>fetch</tt> to download messages and process them.
+
+  @fetcher.fetch
+  
+== Configuration
+
+The following options can be passed to the <tt>Fetcher.create</tt> factory method:
+
+[<tt>type</tt>] POP or IMAP 
+[<tt>server</tt>] The IP address or domain name of the server 
+[<tt>port</tt>] The port to connect to (defaults to the standard port for the type of server) 
+[<tt>ssl</tt>] Set to any value to use SSL encryption 
+[<tt>username</tt>] The username used to connect to the server 
+[<tt>password</tt>] The password used to connect to the server
+[<tt>authentication</tt>] The authentication scheme to use (IMAP only). Supports LOGIN, CRAM-MD5, and PASSWORD (defaults to PLAIN) 
+[<tt>use_login</tt>] Set to any value to use the LOGIN command instead of AUTHENTICATE. Some servers, like GMail, do not support AUTHENTICATE (IMAP only).
+[<tt>sleep_time</tt>] The number of seconds to sleep between fetches (defaults to 60 seconds; valid only for the generated daemon) 
+[<tt>processed_folder</tt>] The name of a folder to move mail to after it has been processed (IMAP only). <b>NOTE:</b> If not specified, mail is deleted. 
+[<tt>error_folder</tt>] The name a folder to move mail that causes an error during processing (IMAP only). Defaults to +bogus+.
+
+== Running via cron
+
+You can also run the Fetcher periodically via cron. It is important to ensure that only one 
+instance is running at one time, and for that the {Lockfile gem}[http://codeforpeople.com/lib/ruby/lockfile/] is recommended.
+
+Here is an example script to be with <tt>script/runner</tt> via cron:
+
+  begin
+    Lockfile.new('cron_mail_fetcher.lock', :retries => 0) do
+      config = YAML.load_file("#{RAILS_ROOT}/config/mail.yml")
+      config = config[RAILS_ENV].to_options
+
+      fetcher = Fetcher.create({:receiver => MailReceiver}.merge(config))
+      fetcher.fetch
+    end
+  rescue Lockfile::MaxTriesLockError => e
+    puts "Another fetcher is already running. Exiting."
+  end
+
+== Extending
+
+You can subclass <tt>Fetcher::Base</tt> or one of the protocol-specific classed to override the standard behavior.
+
+== Further documentation
+
+<shameless-plug>
+
+You can read more about how to use the Fetcher in the PeepCode book {Receiving Email with Ruby}[https://peepcode.com/products/mms2r-pdf].
+
+</shameless-plug>
+
+== Credit & Copyright
+
+Created by Dan Weinand and Luke Francl. Development supported by {Slantwise Design}[http://slantwisedesign.com].
+
+Licensed under the terms of the MIT License. Be excellent to each other.

data/Rakefile

@@ -0,0 +1,30 @@
+require 'rake'
+require 'rake/testtask'
+require 'rake/rdoctask'
+require 'gemspec'
+
+desc 'Default: run unit tests.'
+task :default => :test
+
+desc "Generate gemspec"
+task :gemspec do
+  File.open("#{Dir.pwd}/#{GEM_NAME}.gemspec", 'w') do |f|
+    f.write(GEM_SPEC.to_ruby)
+  end
+end
+
+desc 'Test the fetcher plugin.'
+Rake::TestTask.new(:test) do |t|
+  t.libs << 'lib'
+  t.pattern = 'test/**/*_test.rb'
+  t.verbose = true
+end
+
+desc 'Generate documentation for the fetcher plugin.'
+Rake::RDocTask.new(:rdoc) do |rdoc|
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title    = 'Fetcher'
+  rdoc.options << '--line-numbers' << '--inline-source'
+  rdoc.rdoc_files.include('README')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end

data/fetcher.gemspec

@@ -0,0 +1,27 @@
+# -*- encoding: utf-8 -*-
+
+Gem::Specification.new do |s|
+  s.name = %q{fetcher}
+  s.version = "0.1.1"
+
+  s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
+  s.authors = ["Luke Francl"]
+  s.date = %q{2009-06-27}
+  s.email = %q{look@recursion.org}
+  s.extra_rdoc_files = ["README.rdoc"]
+  s.files = ["fetcher.gemspec", "gemspec.rb", "init.rb", "lib", "lib/fetcher", "lib/fetcher/base.rb", "lib/fetcher/imap.rb", "lib/fetcher/pop.rb", "lib/fetcher.rb", "lib/vendor", "lib/vendor/plain_imap.rb", "lib/vendor/secure_pop.rb", "MIT-LICENSE", "Rakefile", "README.rdoc", "test", "test/fetcher_test.rb"]
+  s.homepage = %q{http://github.com/winton/fetcher}
+  s.require_paths = ["lib"]
+  s.rubygems_version = %q{1.3.1}
+  s.summary = %q{download email from POP3 or IMAP and do stuff with it. gemified fork}
+
+  if s.respond_to? :specification_version then
+    current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION
+    s.specification_version = 2
+
+    if Gem::Version.new(Gem::RubyGemsVersion) >= Gem::Version.new('1.2.0') then
+    else
+    end
+  else
+  end
+end

data/gemspec.rb

@@ -0,0 +1,17 @@
+GEM_NAME = 'fetcher'
+GEM_FILES = FileList['**/*'] - FileList['coverage', 'coverage/**/*', 'pkg', 'pkg/**/*']
+GEM_SPEC = Gem::Specification.new do |s|
+  # == CONFIGURE ==
+  s.author = "Luke Francl"
+  s.email = "look@recursion.org"
+  s.homepage = "http://github.com/winton/#{GEM_NAME}"
+  s.summary = "download email from POP3 or IMAP and do stuff with it. gemified fork"
+  # == CONFIGURE ==
+  s.extra_rdoc_files = [ "README.rdoc" ]
+  s.files = GEM_FILES.to_a
+  s.has_rdoc = false
+  s.name = GEM_NAME
+  s.platform = Gem::Platform::RUBY
+  s.require_path = "lib"
+  s.version = "0.1.1"
+end

data/lib/fetcher.rb

@@ -0,0 +1,22 @@
+module Fetcher
+  # Use factory-style initialization or insantiate directly from a subclass
+  #
+  # Options:
+  # * <tt>:type</tt> - Name of class as a symbol to instantiate
+  #
+  # Other options are the same as Fetcher::Base.new
+  # 
+  # Example:
+  # 
+  # Fetcher.create(:type => :pop) is equivalent to
+  # Fetcher::Pop.new()
+  def self.create(options={})
+    klass = options.delete(:type)
+    raise ArgumentError, 'Must supply a type' unless klass
+    module_eval "#{klass.to_s.capitalize}.new(options)"
+  end
+end
+
+require 'fetcher/base'
+require 'fetcher/pop'
+require 'fetcher/imap'

data/lib/fetcher/base.rb

@@ -0,0 +1,63 @@
+module Fetcher
+  class Base
+    # Options:
+    # * <tt>:server</tt> - Server to connect to.
+    # * <tt>:username</tt> - Username to use when connecting to server.
+    # * <tt>:password</tt> - Password to use when connecting to server.
+    # * <tt>:receiver</tt> - Receiver object to pass messages to. Assumes the
+    # receiver object has a receive method that takes a message as it's argument
+    #
+    # Additional protocol-specific options implimented by sub-classes
+    #
+    # Example: 
+    #   Fetcher::Base.new(:server => 'mail.example.com',
+    #                     :username => 'pam',
+    #                     :password => 'test',
+    #                     :receiver => IncomingMailHandler)
+    def initialize(options={})
+      %w(server username password receiver).each do |opt|
+        raise ArgumentError, "#{opt} is required" unless options[opt.to_sym]
+        # convert receiver to a Class if it isn't already.
+        if opt == "receiver" && options[:receiver].is_a?(String)
+          options[:receiver] = Kernel.const_get(options[:receiver])
+        end
+          
+        instance_eval("@#{opt} = options[:#{opt}]")
+      end
+    end
+    
+    # Run the fetching process
+    def fetch
+      establish_connection
+      get_messages
+      close_connection
+    end
+    
+    protected
+    
+    # Stub. Should be overridden by subclass.
+    def establish_connection #:nodoc:
+      raise NotImplementedError, "This method should be overridden by subclass"
+    end
+    
+    # Stub. Should be overridden by subclass.
+    def get_messages #:nodoc:
+      raise NotImplementedError, "This method should be overridden by subclass"
+    end
+    
+    # Stub. Should be overridden by subclass.
+    def close_connection #:nodoc:
+      raise NotImplementedError, "This method should be overridden by subclass"
+    end
+    
+    # Send message to receiver object
+    def process_message(message)
+      @receiver.receive(message)
+    end
+    
+    # Stub. Should be overridden by subclass.
+    def handle_bogus_message(message) #:nodoc:
+      raise NotImplementedError, "This method should be overridden by subclass"
+    end
+  end
+end

data/lib/fetcher/imap.rb

@@ -0,0 +1,79 @@
+require File.dirname(__FILE__) + '/../vendor/plain_imap'
+
+module Fetcher
+  class Imap < Base
+    
+    PORT = 143
+    
+    protected
+    
+    # Additional Options:
+    # * <tt>:authentication</tt> - authentication type to use, defaults to PLAIN
+    # * <tt>:port</tt> - port to use (defaults to 143)
+    # * <tt>:ssl</tt> - use SSL to connect
+    # * <tt>:use_login</tt> - use LOGIN instead of AUTHENTICATE to connect (some IMAP servers, like GMail, do not support AUTHENTICATE)
+    # * <tt>:processed_folder</tt> - if set to the name of a mailbox, messages will be moved to that mailbox instead of deleted after processing. The mailbox will be created if it does not exist.
+    # * <tt>:error_folder:</tt> - the name of a mailbox where messages that cannot be processed (i.e., your receiver throws an exception) will be moved. Defaults to "bogus". The mailbox will be created if it does not exist.
+    def initialize(options={})
+      @authentication = options.delete(:authentication) || 'PLAIN'
+      @port = options.delete(:port) || PORT
+      @ssl = options.delete(:ssl)
+      @use_login = options.delete(:use_login)
+      @processed_folder = options.delete(:processed_folder)
+      @error_folder = options.delete(:error_folder) || 'bogus'
+      super(options)
+    end
+    
+    # Open connection and login to server
+    def establish_connection
+      @connection = Net::IMAP.new(@server, @port, @ssl)
+      if @use_login
+        @connection.login(@username, @password)
+      else
+        @connection.authenticate(@authentication, @username, @password)
+      end
+    end
+    
+    # Retrieve messages from server
+    def get_messages
+      @connection.select('INBOX')
+      @connection.uid_search(['ALL']).each do |uid|
+        msg = @connection.uid_fetch(uid,'RFC822').first.attr['RFC822']
+        begin
+          process_message(msg)
+          add_to_processed_folder(uid) if @processed_folder
+        rescue
+          handle_bogus_message(msg)
+        end
+        # Mark message as deleted 
+        @connection.uid_store(uid, "+FLAGS", [:Seen, :Deleted])
+      end
+    end
+    
+    # Store the message for inspection if the receiver errors
+    def handle_bogus_message(message)
+      create_mailbox(@error_folder)
+      @connection.append(@error_folder, message)
+    end
+    
+    # Delete messages and log out
+    def close_connection
+      @connection.expunge
+      @connection.logout
+      @connection.disconnect
+    rescue
+    end
+    
+    def add_to_processed_folder(uid)
+      create_mailbox(@processed_folder)
+      @connection.uid_copy(uid, @processed_folder)
+    end
+    
+    def create_mailbox(mailbox)
+      unless @connection.list("", mailbox)
+        @connection.create(mailbox)
+      end
+    end
+    
+  end
+end

data/lib/fetcher/pop.rb

@@ -0,0 +1,50 @@
+require File.dirname(__FILE__) + '/../vendor/secure_pop'
+
+module Fetcher
+  class Pop < Base
+
+    protected
+    
+    # Additional Options:
+    # * <tt>:ssl</tt> - whether or not to use ssl encryption
+    # * <tt>:port</tt> - port to use (defaults to 110)
+    def initialize(options={})
+      @ssl = options.delete(:ssl)
+      @port = options.delete(:port) || Net::POP3.default_port
+      super(options)
+    end
+    
+    # Open connection and login to server
+    def establish_connection
+      @connection = Net::POP3.new(@server, @port)
+      @connection.enable_ssl(OpenSSL::SSL::VERIFY_NONE) if @ssl
+      @connection.start(@username, @password)
+    end
+    
+    # Retrieve messages from server
+    def get_messages
+      unless @connection.mails.empty?
+        @connection.each_mail do |msg|
+          begin
+            process_message(msg.pop)
+          rescue
+            handle_bogus_message(msg.pop)
+          end
+          # Delete message from server
+          msg.delete
+        end
+      end
+    end
+    
+    # Store the message for inspection if the receiver errors
+    def handle_bogus_message(message)
+      # This needs a good solution
+    end
+    
+    # Close connection to server
+    def close_connection
+      @connection.finish
+    end
+    
+  end
+end

data/lib/vendor/plain_imap.rb

@@ -0,0 +1,21 @@
+require 'net/imap'
+# add plain as an authentication type...
+# This is taken from:
+# http://svn.ruby-lang.org/cgi-bin/viewvc.cgi/trunk/lib/net/imap.rb?revision=7657&view=markup&pathrev=10966
+
+# Authenticator for the "PLAIN" authentication type.  See
+# #authenticate().
+class PlainAuthenticator
+  def process(data)
+    return "\0#{@user}\0#{@password}"
+  end
+
+  private
+
+  def initialize(user, password)
+    @user = user
+    @password = password
+  end
+end
+
+Net::IMAP.add_authenticator "PLAIN", PlainAuthenticator

data/lib/vendor/secure_pop.rb

@@ -0,0 +1,125 @@
+require 'socket'
+require 'net/pop'
+require 'net/protocol'
+require 'openssl/ssl'
+
+# Backport of ruby 1.9's POP3 SSL support
+class Net::POP3
+  @@usessl = nil
+  @@verify = nil
+  @@certs = nil
+  PORT = 110
+  SSL_PORT = 995
+  
+  def self.default_port
+    PORT
+  end
+  
+  def self.default_ssl_port
+    SSL_PORT
+  end
+  
+  # Enable SSL for all new instances.
+  # +verify+ is the type of verification to do on the Server Cert; Defaults
+  # to OpenSSL::SSL::VERIFY_PEER.
+  # +certs+ is a file or directory holding CA certs to use to verify the 
+  # server cert; Defaults to nil.
+  def self.enable_ssl( verify = OpenSSL::SSL::VERIFY_PEER, certs = nil )
+    @@usessl = true
+    @@verify = verify
+    @@certs = certs  
+  end
+  
+  # Disable SSL for all new instances.
+  def self.disable_ssl
+    @@usessl = nil
+    @@verify = nil
+    @@certs = nil
+  end
+  
+  # Creates a new POP3 object.
+  # +addr+ is the hostname or ip address of your POP3 server.
+  # The optional +port+ is the port to connect to.
+  # The optional +isapop+ specifies whether this connection is going
+  # to use APOP authentication; it defaults to +false+.
+  # This method does *not* open the TCP connection.
+  def initialize(addr, port = nil, isapop = false)
+    @address = addr
+    @usessl = @@usessl
+    if @usessl
+      @port = port || SSL_PORT
+    else
+      @port = port || PORT
+    end
+    @apop = isapop
+    
+    @certs = @@certs
+    @verify = @@verify
+    
+    @command = nil
+    @socket = nil
+    @started = false
+    @open_timeout = 30
+    @read_timeout = 60
+    @debug_output = nil
+
+    @mails = nil
+    @n_mails = nil
+    @n_bytes = nil
+  end
+  
+  # does this instance use SSL?
+  def usessl?
+    @usessl
+  end
+  
+  # Enables SSL for this instance.  Must be called before the connection is
+  # established to have any effect.
+  # +verify+ is the type of verification to do on the Server Cert; Defaults
+  # to OpenSSL::SSL::VERIFY_PEER.
+  # +certs+ is a file or directory holding CA certs to use to verify the 
+  # server cert; Defaults to nil.
+  # +port+ is port to establish the SSL conection on; Defaults to 995.
+  def enable_ssl(verify = OpenSSL::SSL::VERIFY_PEER, certs = nil, 
+                 port = SSL_PORT)
+    @usessl = true
+    @verify = verify
+    @certs = certs
+    @port = port
+  end
+  
+  def disable_ssl
+    @usessl = nil
+    @verify = nil
+    @certs = nil
+  end
+  
+  def do_start( account, password )
+    s = timeout(@open_timeout) { TCPSocket.open(@address, @port) }
+    if @usessl
+      unless defined?(OpenSSL)
+        raise "SSL extension not installed"
+      end
+      sslctx = OpenSSL::SSL::SSLContext.new
+      sslctx.verify_mode = @verify
+      sslctx.ca_file = @certs if @certs && FileTest::file?(@certs)
+      sslctx.ca_path = @certs if @certs && FileTest::directory?(@certs)
+      s = OpenSSL::SSL::SSLSocket.new(s, sslctx)
+      s.sync_close = true
+      s.connect
+    end
+    
+    @socket = Net::InternetMessageIO.new(s)
+    on_connect
+    @command = Net::POP3Command.new(@socket)
+    if apop?
+      @command.apop account, password
+    else
+      @command.auth account, password
+    end
+    @started = true
+  ensure
+    do_finish if not @started
+  end
+  private :do_start
+end

data/test/fetcher_test.rb

@@ -0,0 +1,74 @@
+# require File.dirname(__FILE__) + '/../../../../config/boot'
+require 'rubygems'
+require 'test/unit'
+require 'mocha'
+require 'fetcher'
+
+class FetcherTest < Test::Unit::TestCase
+  
+  def setup
+    @receiver = mock()
+  end
+  
+  def test_should_set_configuration_instance_variables
+    create_fetcher
+    assert_equal 'test.host', @fetcher.instance_variable_get(:@server)
+    assert_equal 'name', @fetcher.instance_variable_get(:@username)
+    assert_equal 'password', @fetcher.instance_variable_get(:@password)
+    assert_equal @receiver, @fetcher.instance_variable_get(:@receiver)
+  end
+  
+  def test_should_require_subclass
+    create_fetcher
+    assert_raise(NotImplementedError) { @fetcher.fetch }
+  end
+  
+  def test_should_require_server
+    assert_raise(ArgumentError) { create_fetcher(:server => nil) }
+  end
+  
+  def test_should_require_username
+    assert_raise(ArgumentError) { create_fetcher(:username => nil) }
+  end
+  
+  def test_should_require_password
+    assert_raise(ArgumentError) { create_fetcher(:password => nil) }
+  end
+  
+  def test_should_require_receiver
+    assert_raise(ArgumentError) { create_fetcher(:receiver => nil) }
+  end
+  
+  def create_fetcher(options={})
+    @fetcher = Fetcher::Base.new({:server => 'test.host', :username => 'name', :password => 'password', :receiver => @receiver}.merge(options))
+  end
+  
+end
+
+class FactoryFetcherTest < Test::Unit::TestCase
+  
+  def setup
+    @receiver = mock()
+    @pop_fetcher = Fetcher.create(:type => :pop, :server => 'test.host',
+                               :username => 'name',
+                               :password => 'password',
+                               :receiver => @receiver)
+    
+  @imap_fetcher = Fetcher.create(:type => :imap, :server => 'test.host',
+                              :username => 'name',
+                              :password => 'password',
+                              :receiver => @receiver)
+  end
+  
+  def test_should_be_sublcass
+    assert_equal Fetcher::Pop, @pop_fetcher.class
+    assert_equal Fetcher::Imap, @imap_fetcher.class
+  end
+  
+  def test_should_require_type
+    assert_raise(ArgumentError) { Fetcher.create({}) }
+  end
+  
+end
+
+# Write tests for sub-classes

metadata

@@ -0,0 +1,69 @@
+--- !ruby/object:Gem::Specification 
+name: winton-fetcher
+version: !ruby/object:Gem::Version 
+  version: 0.1.1
+platform: ruby
+authors: 
+- Luke Francl
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2009-06-27 00:00:00 -07:00
+default_executable: 
+dependencies: []
+
+description: 
+email: look@recursion.org
+executables: []
+
+extensions: []
+
+extra_rdoc_files: 
+- README.rdoc
+files: 
+- fetcher.gemspec
+- gemspec.rb
+- init.rb
+- lib
+- lib/fetcher
+- lib/fetcher/base.rb
+- lib/fetcher/imap.rb
+- lib/fetcher/pop.rb
+- lib/fetcher.rb
+- lib/vendor
+- lib/vendor/plain_imap.rb
+- lib/vendor/secure_pop.rb
+- MIT-LICENSE
+- Rakefile
+- README.rdoc
+- test
+- test/fetcher_test.rb
+has_rdoc: false
+homepage: http://github.com/winton/fetcher
+post_install_message: 
+rdoc_options: []
+
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.2.0
+signing_key: 
+specification_version: 2
+summary: download email from POP3 or IMAP and do stuff with it. gemified fork
+test_files: []
+