larch 1.1.0.dev.20091126 → 1.1.0.dev.20091203

data/README.rdoc

@@ -22,7 +22,7 @@ Latest stable release:
 
 Latest development version:
 
-  gem install larch -s http://gemcutter.org --prerelease
+  gem install larch --pre
 
 == Usage
 
@@ -213,8 +213,47 @@ servers:
 The following servers do not work well with Larch:
 
 * BlitzMail - Buggy server implementation; fails to properly quote or escape
-  some IMAP responses, which can cause Larch to hang waiting for a terminating
-  character that will never arrive.
+  some IMAP responses, which can cause Net::IMAP to hang waiting for a
+  terminating character that will never arrive.
+
+=== Gmail Quirks
+
+Gmail's IMAP implementation is quirky. Larch does its best to work around these
+quirks whenever possible, but here are a few things to watch out for:
+
+==== "Some messages could not be FETCHed" error
+
+This error indicates that a message on Gmail is corrupt, and Gmail itself is
+unable to read it. The message will continue to show up in the mailbox, but all
+attempts to access it via IMAP, POP, or the Gmail web interface will result in
+errors. Larch will try to skip these messages and continue processing others
+if possible.
+
+It's not clear how this corruption occurs or exactly what kind of corruption
+causes these errors, although in every case I'm aware of, the corrupt message
+has originated outside of Gmail (Gmail itself does not corrupt the message).
+There is currently no known solution for this problem apart from deleting the
+corrupted messages.
+
+==== Maximum folder name length of 40 characters
+
+Folders in Gmail IMAP are really just labels, and a nested folder hierarchy like
+"Foo\Bar\Baz" is actually just a single label with "\" characters in it. Since
+Gmail limits label names to 40 characters, it's impossible to create a nested
+folder hierarchy with a combined name length of more than 40 characters
+including delimiters.
+
+If you try to copy a too-long folder hierarchy to Gmail using Larch, Gmail will
+return a "Folder name is not allowed" error. To work around this, reorganize
+your mail into a shallower hierarchy on the source server to avoid hitting the
+folder name length limit.
+
+==== Folder names cannot contain leading or trailing whitespace
+
+Most IMAP servers allow folder names to contain leading and trailing whitespace,
+such as "  folder  ". Gmail does not. When copying folders to Gmail, Larch will
+automatically remove leading and trailing whitespace in folder names to prevent
+errors.
 
 == Known Issues
 
@@ -223,10 +262,14 @@ The following servers do not work well with Larch:
   cause a deadlock to occur if a connection drops unexpectedly (either due to
   network issues or because the server closed the connection without warning)
   when the server has already begun sending a response and Net::IMAP is
-  waiting to receive more data.
+  waiting to receive more data. If this happens, Net::IMAP will continue waiting
+  forever without passing control back to Larch, and you will need to manually
+  kill and restart Larch.
 
-  If this happens, Net::IMAP will continue waiting forever without passing
-  control back to Larch, and you will need to manually kill and restart Larch.
+  Net::IMAP in Ruby 1.8 has also been known to hang when it can't parse a server
+  response, either because the response itself is malformed or because of a
+  bug in Net::IMAP's parser. This is rare, but it happens. Unfortunately there's
+  nothing Larch can do about this.
 
 * The Ruby package on Debian, Ubuntu, and some other Debian-based Linux
   distributions doesn't include the OpenSSL standard library. If you see an
@@ -241,6 +284,9 @@ The Larch mailing list is the best place for questions, comments, and discussion
 about Larch. You can join the list or view the archives at
 http://groups.google.com/group/larch
 
+First-time senders to the list are moderated to prevent spam, so there may be a
+delay before your first message shows up.
+
 == Credit
 
 The Larch::IMAP class borrows heavily from Sup[http://sup.rubyforge.org] by

data/lib/larch/config.rb

@@ -95,6 +95,8 @@ class Config
       @cached['from-folder'] ||= 'INBOX'
       @cached['to-folder']   ||= 'INBOX'
     end
+
+    @cached['exclude'].flatten!
   end
 
   private

data/lib/larch/imap/mailbox.rb

@@ -16,17 +16,15 @@ class Mailbox
   def initialize(imap, name, delim, subscribed, *attr)
     raise ArgumentError, "must provide a Larch::IMAP instance" unless imap.is_a?(Larch::IMAP)
 
+    @attr       = attr.flatten
+    @delim      = delim
+    @flags      = []
     @imap       = imap
+    @last_scan  = nil
     @name       = name
     @name_utf7  = Net::IMAP.encode_utf7(@name)
-    @delim      = delim
-    @flags      = []
     @perm_flags = []
     @subscribed = subscribed
-    @attr       = attr.flatten
-
-    @last_scan = nil
-    @mutex     = Mutex.new
 
     # Valid mailbox states are :closed (no mailbox open), :examined (mailbox
     # open and read-only), or :selected (mailbox open and read-write).
@@ -52,6 +50,8 @@ class Mailbox
     # Create private convenience methods (debug, info, warn, etc.) to make
     # logging easier.
     Logger::LEVELS.each_key do |level|
+      next if Mailbox.private_method_defined?(level)
+
       Mailbox.class_eval do
         define_method(level) do |msg|
           Larch.log.log(level, "#{@imap.username}@#{@imap.host}: #{@name}: #{msg}")
@@ -82,7 +82,7 @@ class Mailbox
         next true if flag == :Recent
 
         unless @flags.include?(flag) || @perm_flags.include?(:*) || @perm_flags.include?(flag)
-          debug "flag not supported on destination: #{flag}"
+          info "flag not supported on destination: #{flag}"
           true
         end
       end
@@ -167,14 +167,15 @@ class Mailbox
 
   # Resets the mailbox state.
   def reset
-    @mutex.synchronize { @state = :closed }
+    @state = :closed
   end
 
   # Fetches message headers from this mailbox.
   def scan
-    return if @last_scan && (Time.now - @last_scan) < SCAN_INTERVAL
+    now = Time.now.to_i
+    return if @last_scan && (now - @last_scan) < SCAN_INTERVAL
     first_scan = @last_scan.nil?
-    @mutex.synchronize { @last_scan = Time.now }
+    @last_scan = now
 
     # Compare the mailbox's current status with its last known status.
     begin
@@ -314,7 +315,8 @@ class Mailbox
       rescue => e
         # Set this mailbox's uidnext value to the last known good UID that was
         # stored in the database, plus 1. This will allow Larch to resume where
-        # the error occurred on the next attempt rather than having to start over.
+        # the error occurred on the next attempt rather than having to start
+        # over.
         @db_mailbox.update(:uidnext => last_good_uid + 1) if last_good_uid
         raise
       end
@@ -329,7 +331,7 @@ class Mailbox
     return false if subscribed? && !force
 
     @imap.safely { @imap.conn.subscribe(@name_utf7) } unless @imap.options[:dry_run]
-    @mutex.synchronize { @subscribed = true }
+    @subscribed = true
     @db_mailbox.update(:subscribed => 1)
 
     true
@@ -345,7 +347,7 @@ class Mailbox
     return false unless subscribed? || force
 
     @imap.safely { @imap.conn.unsubscribe(@name_utf7) } unless @imap.options[:dry_run]
-    @mutex.synchronize { @subscribed = false }
+    @subscribed = false
     @db_mailbox.update(:subscribed => 0)
 
     true
@@ -397,13 +399,13 @@ class Mailbox
 
     @imap.safely do
       begin
-        @mutex.synchronize { @state = :closed }
+        @state = :closed
 
         debug "examining mailbox"
         @imap.conn.examine(@name_utf7)
         refresh_flags
 
-        @mutex.synchronize { @state = :examined }
+        @state = :examined
 
       rescue Net::IMAP::NoResponseError => e
         raise Error, "unable to examine mailbox: #{e.message}"
@@ -425,13 +427,13 @@ class Mailbox
 
     @imap.safely do
       begin
-        @mutex.synchronize { @state = :closed }
+        @state = :closed
 
         debug "selecting mailbox"
         @imap.conn.select(@name_utf7)
         refresh_flags
 
-        @mutex.synchronize { @state = :selected }
+        @state = :selected
 
       rescue Net::IMAP::NoResponseError => e
         raise Error, "unable to select mailbox: #{e.message}" unless create

data/lib/larch/imap.rb

@@ -56,7 +56,6 @@ class IMAP
 
     @conn      = nil
     @mailboxes = {}
-    @mutex     = Mutex.new
     @quirks    = {
       :gmail => false
     }
@@ -69,6 +68,8 @@ class IMAP
     # Create private convenience methods (debug, info, warn, etc.) to make
     # logging easier.
     Logger::LEVELS.each_key do |level|
+      next if IMAP.private_method_defined?(level)
+
       IMAP.class_eval do
         define_method(level) do |msg|
           Larch.log.log(level, "#{username}@#{host}: #{msg}")
@@ -185,7 +186,7 @@ class IMAP
 
       raise unless (retries += 1) <= @options[:max_retries]
 
-      info "#{e.class.name}: #{e.message} (reconnecting)"
+      warn "#{e.class.name}: #{e.message} (reconnecting)"
 
       reset
       sleep 1 * retries
@@ -198,7 +199,7 @@ class IMAP
 
       raise unless (retries += 1) <= @options[:max_retries]
 
-      info "#{e.class.name}: #{e.message} (will retry)"
+      warn "#{e.class.name}: #{e.message} (will retry)"
 
       sleep 1 * retries
       retry
@@ -252,10 +253,8 @@ class IMAP
 
   # Resets the connection and mailbox state.
   def reset
-    @mutex.synchronize do
-      @conn = nil
-      @mailboxes.each_value {|mb| mb.reset }
-    end
+    @conn = nil
+    @mailboxes.each_value {|mb| mb.reset }
   end
 
   def safe_connect
@@ -277,7 +276,7 @@ class IMAP
       # verification errors.
       raise if e.is_a?(OpenSSL::SSL::SSLError) && e.message =~ /certificate verify failed/
 
-      info "#{e.class.name}: #{e.message} (will retry)"
+      warn "#{e.class.name}: #{e.message} (will retry)"
 
       reset
       sleep 1 * retries
@@ -289,7 +288,7 @@ class IMAP
   end
 
   def unsafe_connect
-    info "connecting..."
+    debug "connecting..."
 
     exception = nil
 
@@ -322,7 +321,7 @@ class IMAP
             @conn.authenticate(method, username, password)
           end
 
-          info "authenticated using #{method}"
+          debug "authenticated using #{method}"
 
         rescue Net::IMAP::BadResponseError, Net::IMAP::NoResponseError => e
           debug "#{method} auth failed: #{e.message}"
@@ -343,18 +342,16 @@ class IMAP
     all        = safely { @conn.list('', '*') } || []
     subscribed = safely { @conn.lsub('', '*') } || []
 
-    @mutex.synchronize do
-      # Remove cached mailboxes that no longer exist.
-      @mailboxes.delete_if {|k, v| !all.any?{|mb| Net::IMAP.decode_utf7(mb.name) == k}}
+    # Remove cached mailboxes that no longer exist.
+    @mailboxes.delete_if {|k, v| !all.any?{|mb| Net::IMAP.decode_utf7(mb.name) == k}}
 
-      # Update cached mailboxes.
-      all.each do |mb|
-        name = Net::IMAP.decode_utf7(mb.name)
-        name = 'INBOX' if name.downcase == 'inbox'
+    # Update cached mailboxes.
+    all.each do |mb|
+      name = Net::IMAP.decode_utf7(mb.name)
+      name = 'INBOX' if name.downcase == 'inbox'
 
-        @mailboxes[name] ||= Mailbox.new(self, name, mb.delim,
-            subscribed.any?{|s| s.name == mb.name}, mb.attr)
-      end
+      @mailboxes[name] ||= Mailbox.new(self, name, mb.delim,
+          subscribed.any?{|s| s.name == mb.name}, mb.attr)
     end
 
     # Remove mailboxes that no longer exist from the database.

data/lib/larch/version.rb

@@ -1,6 +1,6 @@
 module Larch
   APP_NAME      = 'Larch'
-  APP_VERSION   = '1.1.0.dev.20091126'
+  APP_VERSION   = '1.1.0.dev.20091203'
   APP_AUTHOR    = 'Ryan Grove'
   APP_EMAIL     = 'ryan@wonko.com'
   APP_URL       = 'http://github.com/rgrove/larch/'

data/lib/larch.rb

@@ -1,7 +1,3 @@
-# Prepend this file's directory to the include path if it's not there already.
-$:.unshift(File.dirname(File.expand_path(__FILE__)))
-$:.uniq!
-
 require 'cgi'
 require 'digest/md5'
 require 'fileutils'
@@ -23,7 +19,7 @@ require 'larch/version'
 module Larch
 
   class << self
-    attr_reader :config, :db, :log, :exclude
+    attr_reader :config, :db, :exclude, :log
 
     EXCLUDE_COMMENT = /#.*$/
     EXCLUDE_REGEX   = /^\s*\/(.*)\/\s*/
@@ -37,15 +33,7 @@ module Larch
       @log    = Logger.new(@config[:verbosity])
       @db     = open_db(@config[:database])
 
-      @exclude = @config[:exclude].map do |e|
-        if e =~ EXCLUDE_REGEX
-          Regexp.new($1, Regexp::IGNORECASE)
-        else
-          glob_to_regex(e.strip)
-        end
-      end
-
-      load_exclude_file(@config[:exclude_file]) if @config[:exclude_file]
+      parse_exclusions
 
       Net::IMAP.debug = true if @log.level == :insane
 
@@ -256,6 +244,17 @@ module Larch
       raise Larch::IMAP::FatalError, "error in exclude file at line #{lineno}: #{e}"
     end
 
+    def parse_exclusions
+      @exclude = @config[:exclude].map do |e|
+        if e =~ EXCLUDE_REGEX
+          Regexp.new($1, Regexp::IGNORECASE)
+        else
+          glob_to_regex(e.strip)
+        end
+      end
+
+      load_exclude_file(@config[:exclude_file]) if @config[:exclude_file]
+    end
   end
 
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: larch
 version: !ruby/object:Gem::Version 
-  version: 1.1.0.dev.20091126
+  version: 1.1.0.dev.20091203
 platform: ruby
 authors: 
 - Ryan Grove
@@ -9,7 +9,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2009-11-26 00:00:00 -08:00
+date: 2009-12-03 00:00:00 -08:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency