mail 2.2.0 → 2.2.1

data/CHANGELOG.rdoc

@@ -1,3 +1,21 @@
+== Thu May 13 02:49:47 UTC 2010 Mikel Lindsaar <raasdnil@gmail.com>
+
+* Fixed Ruby 1.8.6 and 1.9.x incompatibilities because we were using each_with_index and each_line
+* Redid the folding to make it much much faster for long strings (the old one really crumbled with headers longer than 100,000 characters (Lars Pind <lars@pinds.com>)
+* Mail now only returns one encoded-word per line (Lars Pind <lars@pinds.com>)
+* Fixed all the previous issues with extra white-space and an extra = at the end of the encoded-words. (Lars Pind <lars@pinds.com>)
+* Make sure we can handle decoding of very long strings efficiently (Lars Pind <lars@pinds.com>)
+* Handle setting of charset through []= method, so we don't get the warning (Lars Pind <lars@pinds.com>)
+* Remove the trailing =\n that pack('M') adds (Lars Pind <lars@pinds.com>)
+* Handle multiple quoted words in Encodings.unquote_and_convert_to (Eric Kidd <git@randomhacks.net>)
+* Ruby 1.9: mark source encoding so it's usable with -Ks, -Ke, etc (Jeremy Kemper <jeremy@bitsweat.net>)
+* Add #include? to mail body for convenience (Maxim Chernyak <max@bitsonnet.com>)
+* Use Mail::TestMailer.deliveries in README example (John Trupiano <jtrupiano@gmail.com>)
+* Allow bundler to automatically build a gem directly from git (Eric Kidd <git@randomhacks.net>)
+* Added recursive parsing of attachments inside message/rfc822 parts (Ubiratan Pires Alberton <u.alberton@gmail.com>)
+* Adding #inline_content_id for attachments and parts (mikel)
+* Updating readme (mikel)
+
 == Sun Apr 11 07:49:15 UTC 2010 Mikel Lindsaar <raasdnil@gmail.com>
 
 * Lots of updates on encoding and decoding of headers and unstructured fields.  This

data/README.rdoc

@@ -27,17 +27,15 @@ you are doing, you can fiddle with every last bit of your email directly.
 
 Mail is tested and works on the following platforms:
 
-* ree-1.8.7-2009.10 [ x86_64 ]
-* ruby-1.8.6-p383 [ x86_64 ]
-* ruby-1.8.7-p248 [ x86_64 ]
-* ruby-1.9.1-head [ x86_64 ]
-* ruby-1.9.1-p376 [ x86_64 ]
+* jruby-1.4.0 [ [x86_64-java] ]
+* ree-1.8.7-2010.01 [ x86_64 ]
+* ruby-1.8.7-p249 [ x86_64 ]
+* ruby-1.9.1-head [ x86_64 ] (10 April 2010)
+* ruby-1.9.2-head [ x86_64 ] (10 April 2010)
 
-Mail seems to work fine on JRuby as well, however, the Base64 and Quoted-Printable
-encoding methods are implemented differently and return different values so 6 
-specs fail.  This needs to be triple checked if all OK.
-
-* jruby-1.4.0 [x86_64-java]
+Mail seems to work fine on JRuby as well, however there is some weird QP
+problem that only shows up while running the specs, running it directly
+works fine.... weird.  Any jRuby gurus who want to help out, please do.
 
 == Discussion
 
@@ -52,6 +50,8 @@ the Google Group http://groups.google.com/group/mail-ruby
 * Support for reading multipart/report emails & getting details from such
 * Support for multibyte emails - needs quite a lot of work and testing
 * Wrappers for File, Net/POP3, Net/SMTP
+* Auto encoding of non US-ASCII header fields
+* Auto encoding of non US-ASCII bodies
 
 Mail is RFC2822 compliant now, that is, it can parse and generate valid US-ASCII
 emails.  There are a few obsoleted syntax emails that it will have problems with, but
@@ -108,7 +108,7 @@ may or may not be a problem for you.
 
 If you want to install mail manually, you can download the gem from github and do:
 
- # gem install mail-1.2.1.gem
+ # gem install mail-2.2.0.gem
  
 == Encodings
 
@@ -512,7 +512,7 @@ sending emails, the TestMailer can do this for you.
     delivery_method :test
   end
   => #<Mail::Configuration:0x19345a8 @delivery_method=Mail::TestMailer>
-  Mail.deliveries
+  Mail::TestMailer.deliveries
   => []
   Mail.deliver do
     to 'mikel@me.com'
@@ -521,11 +521,11 @@ sending emails, the TestMailer can do this for you.
     body 'hello'
   end
   => #<Mail::Message:0x19284ec ...
-  Mail.deliveries.length
+  Mail::TestMailer.deliveries.length
   => 1
-  Mail.deliveries.first
+  Mail::TestMailer.deliveries.first
   => #<Mail::Message:0x19284ec ...
-  Mail.deliveries.clear
+  Mail::TestMailer.deliveries.clear
   => []
 
 == Excerpts from TREC Spam Corpus 2005

data/Rakefile

@@ -1,34 +1,13 @@
 require File.expand_path('../spec/environment', __FILE__)
 
 require 'rake/rdoctask'
-require 'rake/gempackagetask'
 require 'rake/testtask'
 require 'spec/rake/spectask'
 require 'cucumber/rake/task'
 
-spec = Gem::Specification.new do |s|
-  s.name        = "mail"
-  s.version     = "2.2.0"
-  s.author      = "Mike Lindsaar"
-  s.email       = "raasdnil@gmail.com"
-  s.homepage    = "http://github.com/mikel/mail"
-  s.description = "A really Ruby Mail handler."
-  s.summary     = "Mail provides a nice Ruby DSL for making, sending and reading emails."
-
-  s.platform = Gem::Platform::RUBY
-  s.has_rdoc = true
-  s.extra_rdoc_files = ["README.rdoc", "CHANGELOG.rdoc", "TODO.rdoc"]
-
-  s.add_dependency('activesupport', ">= 2.3.4")
-  s.add_dependency('mime-types')
-  s.add_dependency('treetop', '>= 1.4.5')
-
-  s.require_path = 'lib'
-  s.files = %w(README.rdoc Rakefile TODO.rdoc) + Dir.glob("lib/**/*")
-end
-
-Rake::GemPackageTask.new(spec) do |pkg|
-  pkg.gem_spec = spec
+desc "Build a gem file"
+task :build do
+  system "gem build mail.gemspec"
 end
 
 task :default => :spec
@@ -50,13 +29,6 @@ Spec::Rake::SpecTask.new(:spec) do |t|
   t.libs << "#{File.dirname(__FILE__)}/spec/mail"
 end
 
-Rake::TestTask.new(:test) do |t|
-  t.libs << 'test'
-  t.pattern = 'test/**/tc_*.rb'
-  t.verbose = true
-  t.warning = false
-end
-
 Rake::RDocTask.new(:rdoc) do |rdoc|
   rdoc.rdoc_dir = 'rdoc'
   rdoc.title    = 'Mail'

data/lib/mail/attachments_list.rb

@@ -1,10 +1,12 @@
 module Mail
   class AttachmentsList < Array
-    
+
     def initialize(parts_list)
       @parts_list = parts_list
       parts_list.map { |p|
-        if p.parts.empty?
+        if p.content_type == "message/rfc822"
+          Mail.new(p.body).attachments
+        elsif p.parts.empty?
           p if p.attachment?
         else
           p.attachments
@@ -12,12 +14,12 @@ module Mail
       }.flatten.compact.each { |a| self << a }
       self
     end
-    
+
     # Returns the attachment by filename or at index.
-    # 
+    #
     # mail.attachments['test.png'] = File.read('test.png')
     # mail.attachments['test.jpg'] = File.read('test.jpg')
-    # 
+    #
     # mail.attachments['test.png'].filename #=> 'test.png'
     # mail.attachments[1].filename          #=> 'test.jpg'
     def [](index_value)
@@ -65,10 +67,13 @@ module Mail
           hash[:body].force_encoding("BINARY")
         end
       end
+      
+      attachment = Part.new(hash)
+      attachment.add_content_id(hash[:content_id])
 
-      @parts_list << Part.new(hash)
+      @parts_list << attachment
     end
-   
+
     # Uses the mime type to try and guess the encoding, if it is a binary type, or unknown, then we
     # set it to binary, otherwise as set to plain text
     def guess_encoding
@@ -77,8 +82,8 @@ module Mail
       else
         "binary"
       end
-    end 
- 
+    end
+
     def set_mime_type(filename)
       # Have to do this because MIME::Types is not Ruby 1.9 safe yet
       if RUBY_VERSION >= '1.9'
@@ -88,7 +93,7 @@ module Mail
       end
       @mime_type = MIME::Types.type_for(filename).first
     end
-    
+
   end
 end
 

data/lib/mail/body.rb

@@ -99,6 +99,20 @@ module Mail
       self.decoded.match(regexp)
     end
 
+    # Accepts anything that responds to #to_s and checks if it's a substring of the decoded text
+    #
+    # Examples:
+    #
+    #   body = Mail::Body.new('The body')
+    #   body.include?('The') #=> true
+    #
+    #   body = Mail::Body.new("VGhlIGJvZHk=\n")
+    #   body.encoding = 'base64'
+    #   body.include?('The') #=> true
+    def include?(other)
+      self.decoded.include?(other.to_s)
+    end
+
     # Allows you to set the sort order of the parts, overriding the default sort order.
     # Defaults to 'text/plain', then 'text/enriched', then 'text/html' with any other content
     # type coming after.

data/lib/mail/core_extensions/string.rb

@@ -24,4 +24,16 @@ class String #:nodoc:
   unless method_defined?(:bytesize)
     alias :bytesize :length
   end
+
+  unless ''.respond_to?(:lines)
+    require "enumerator"
+
+    def lines
+      to_a.enum_for(:each)
+    end
+    
+  end
+
 end
+
+

data/lib/mail/encodings.rb

@@ -7,6 +7,7 @@ module Mail
 
   module Encodings
     
+    require "enumerator"
     include Mail::Patterns
     extend  Mail::Utilities
 
@@ -113,24 +114,34 @@ module Mail
     # 
     # String has to be of the format =?<encoding>?[QB]?<string>?=
     def Encodings.value_decode(str)
-      str = str.gsub(/\?=(\s*)=\?/, '?==?') # Remove whitespaces between 'encoded-word's
-
-      # Join QP encoded-words that are adjacent to avoid decoding partial chars
-      str.gsub!( /=\?==\?.+?\?[Qq]\?/m, '' ) if str =~ /\?==\?/
+      # Optimization: If there's no encoded-words in the string, just return it
+      return str unless str.index("=?")
       
-      str.gsub(/(.*?)(=\?.*?\?.\?.*?\?=)|$/m) do # Grab the insides of each encoded-word
-        before = $1.to_s
-        text = $2.to_s
+      str = str.gsub(/\?=(\s*)=\?/, '?==?') # Remove whitespaces between 'encoded-word's
 
-        case
-        when text.to_str =~ /=\?.+\?[Bb]\?/m
-          before + b_value_decode(text)
-        when text.to_str =~ /=\?.+\?[Qq]\?/m
-          before + q_value_decode(text)
+      # Split on white-space boundaries with capture, so we capture the white-space as well
+      str.split(/([ \t])/).map do |text|
+        if text.index('=?') != 0
+          text
         else
-          before + text
+          # Join QP encoded-words that are adjacent to avoid decoding partial chars
+          text.gsub!(/\?\=\=\?.+?\?[Qq]\?/m, '') if text =~ /\?==\?/
+        
+          # Separate encoded-words with a space, so we can treat them one by one
+          text.gsub!(/\?\=\=\?/, '?= =?')
+
+          text.split(/ /).map do |word|
+            case 
+            when word.to_str =~ /=\?.+\?[Bb]\?/m
+              b_value_decode(word)
+            when text.to_str =~ /=\?.+\?[Qq]\?/m
+              q_value_decode(word)
+            else
+              word.to_str
+            end
+          end
         end
-      end
+      end.join("")
     end
     
     # Takes an encoded string of the format =?<encoding>?[QB]?<string>?=
@@ -176,7 +187,8 @@ module Mail
       address.gsub!(/(".*?[^#{us_ascii}].+?")/) { |s| Encodings.b_value_encode(unquote(s), charset) }
       # Then loop through all remaining items and encode as needed
       tokens = address.split(/\s/)
-      tokens.each_with_index.map do |word, i|
+      # Need to use enum_for to stay 1.8.6 compatible
+      tokens.enum_for(:each_with_index).map do |word, i|
         if word.ascii_only?
           word
         else
@@ -199,7 +211,7 @@ module Mail
     def Encodings.b_value_encode(str, encoding = nil)
       return str if str.to_s.ascii_only?
       string, encoding = RubyVer.b_value_encode(str, encoding)
-      string.each_line.map do |str|
+      string.lines.map do |str|
         "=?#{encoding}?B?#{str.chomp}?="
       end.join(" ")
     end
@@ -214,8 +226,8 @@ module Mail
     def Encodings.q_value_encode(str, encoding = nil)
       return str if str.to_s.ascii_only?
       string, encoding = RubyVer.q_value_encode(str, encoding)
-      string.gsub!("=\r\n=", '=') # We already have limited the string to the length we want
-      string.each_line.map do |str|
+      string.gsub!("=\r\n", '') # We already have limited the string to the length we want
+      string.lines.map do |str|
         "=?#{encoding}?Q?#{str.chomp.gsub(/ /, '_')}?="
       end.join(" ")
     end
@@ -243,7 +255,7 @@ module Mail
     end
     
     def Encodings.split_encoding_from_string( str )
-      match = str.match(/\=\?(.+)?\?[QB]\?(.+)?\?\=/mi)
+      match = str.match(/\=\?([^?]+)?\?[QB]\?(.+)?\?\=/mi)
       if match
         [match[1], match[2]]
       else

data/lib/mail/fields/unstructured_field.rb

@@ -102,64 +102,77 @@ module Mail
     #  it is allowed elsewhere.
     def wrapped_value # :nodoc:
       @folded_line = []
-      @unfolded_line = decoded.to_s.clone
+      @unfolded_line = decoded.to_s.split(/[ \t]/)
       fold("#{name}: ".length)
       wrap_lines(name, @folded_line)
     end
-
+   
+    # 6.2. Display of 'encoded-word's
+    # 
+    #  When displaying a particular header field that contains multiple
+    #  'encoded-word's, any 'linear-white-space' that separates a pair of
+    #  adjacent 'encoded-word's is ignored.  (This is to allow the use of
+    #  multiple 'encoded-word's to represent long strings of unencoded text,
+    #  without having to separate 'encoded-word's where spaces occur in the
+    #  unencoded text.)
     def wrap_lines(name, folded_lines)
       result = []
       index = 0
       result[index] = "#{name}: #{folded_lines.shift}"
-      folded_lines.each do |line|
-        if (result[index] + line).length < 77
-          result[index] << " " + line
-        else
-          result[index] << "\r\n\t"
-          index += 1
-          result[index] = line
-        end
-      end
-      result.join
+      result.concat(folded_lines)
+      result.join("\r\n\t")
     end
 
     def fold(prepend = 0) # :nodoc:
-      # Get the last whitespace character, OR we'll just choose 
-      # 78 if there is no whitespace, or 23 for non ascii:
-      # Each QP byte is 6 chars (=0A)
-      # Plus 18 for the =?encoding?Q?= ... ?=
-      # Plus 2 for the \r\n and 1 for the \t
-      # 80 - 2 - 1 - 18 = 59 / 6 ~= 10
-      @unfolded_line.ascii_only? ? (limit = 78 - prepend) : (limit = 10 - prepend)
-      # find the last white space character within the limit
-      if wspp = @unfolded_line.mb_chars.slice(0..limit) =~ /[ \t][^ \t]*$/
-        wrap = true
-        wspp = limit if wspp == 0
-        @folded_line << encode(@unfolded_line.mb_chars.slice!(0...wspp).strip.to_str)
-        @folded_line.flatten!
-      # if no last whitespace before the limit, find the first
-      elsif wspp = @unfolded_line.mb_chars =~ /[ \t][^ \t]/
-        wrap = true
-        wspp = limit if wspp == 0
-        @folded_line << encode(@unfolded_line.mb_chars.slice!(0...wspp).strip.to_str)
-        @folded_line.flatten!
-      # if no whitespace, don't wrap
-      else
-        wrap = false
-      end
-
-      if wrap && @unfolded_line.length > limit
-        fold
-      else
-        @folded_line << encode(@unfolded_line)
-        @folded_line.flatten!
+      encoding = @charset.to_s.upcase.gsub('_', '-')
+      while !@unfolded_line.empty?
+        encoded = false
+        limit = 78 - prepend
+        line = ""
+        while !@unfolded_line.empty?          
+          break unless word = @unfolded_line.first.dup
+          # Remember whether it was non-ascii before we encode it ('cause then we can't tell anymore)
+          non_ascii = word.not_ascii_only?
+          encoded_word = encode(word)
+          # Skip to next line if we're going to go past the limit
+          # Unless this is the first word, in which case we're going to add it anyway
+          # Note: This means that a word that's longer than 998 characters is going to break the spec. Please fix if this is a problem for you.
+          # (The fix, it seems, would be to use encoded-word encoding on it, because that way you can break it across multiple lines and 
+          # the linebreak will be ignored)
+          break if !line.empty? && (line.length + encoded_word.length + 1 > limit)
+          # If word was the first non-ascii word, we're going to make the entire line encoded and we're going to reduce the limit accordingly
+          if non_ascii && !encoded
+            encoded = true
+            encoded_word_safify!(line)
+            limit = limit - 8 - encoding.length  # minus the =?...?Q?...?= part, the possible leading white-space, and the name of the encoding
+          end
+          # Remove the word from the queue ...
+          @unfolded_line.shift
+          # ... add it in encoded form to the current line
+          line << " " unless line.empty?
+          encoded_word_safify!(encoded_word) if encoded
+          line << encoded_word          
+        end
+        # Add leading whitespace if both this and the last line were encoded, because whitespace between two encoded-words is ignored when decoding
+        line = " " + line if encoded && @folded_line.last && @folded_line.last.index('=?') == 0
+        # Encode the line if necessary
+        line = "=?#{encoding}?Q?#{line.gsub(/ /, '_')}?=" if encoded
+        # Add the line to the output and reset the prepend
+        @folded_line << line
+        prepend = 0
       end
     end
-
+        
     def encode(value)
-      value.gsub!("\r", "=0D")
-      value.gsub!("\n", "=0A")
-      Encodings.q_value_encode(value, @charset).split(" ")
+      (value.not_ascii_only? ? [value].pack("M").gsub("=\n", '') : value).gsub("\r", "=0D").gsub("\n", "=0A")
+    end
+    
+    def encoded_word_safify!(value)
+      value.gsub!(/"/,  '=22')
+      value.gsub!(/\(/, '=28')
+      value.gsub!(/\)/, '=29')
+      value.gsub!(/\?/, '=3F')
+      value.gsub!(/_/,  '=5F')
     end
 
   end

data/lib/mail/message.rb

@@ -1180,6 +1180,8 @@ module Mail
         self.body = value
       elsif name.to_s =~ /content[-_]type/i
         header[name] = value
+      elsif name.to_s == 'charset'
+        self.charset = value
       else
         header[name] = value
       end

data/lib/mail/part.rb

@@ -17,7 +17,12 @@ module Mail
     def has_content_id?
       header.has_content_id?
     end
-
+    
+    def inline_content_id
+      add_content_id unless has_content_id?
+      uri_escape(unbracket(content_id))
+    end
+    
     def add_required_fields
       add_content_id unless has_content_id?
       super

data/lib/mail/patterns.rb

@@ -1,3 +1,4 @@
+# encoding: us-ascii
 module Mail
   module Patterns
     white_space = %Q|\x9\x20|

data/lib/mail/utilities.rb

@@ -69,7 +69,7 @@ module Mail
     # 
     # Example:
     # 
-    #  string
+    #  paren( 'This is a string' ) #=> '(This is a string)'
     def paren( str )
       RubyVer.paren( str )
     end
@@ -85,6 +85,26 @@ module Mail
       match ? match[1] : str
     end
     
+    # Wraps a string in angle brackets and escapes any that are in the string itself
+    # 
+    # Example:
+    # 
+    #  bracket( 'This is a string' ) #=> '<This is a string>'
+    def bracket( str )
+      RubyVer.bracket( str )
+    end
+    
+    # Unwraps a string from being wrapped in parenthesis
+    # 
+    # Example:
+    # 
+    #  str = '<This is a string>'
+    #  unbracket( str ) #=> 'This is a string'
+    def unbracket( str )
+      match = str.match(/^\<(.*?)\>$/)
+      match ? match[1] : str
+    end
+    
     # Escape parenthesies in a string
     # 
     # Example:
@@ -95,6 +115,14 @@ module Mail
       RubyVer.escape_paren( str )
     end
     
+    def uri_escape( str )
+      URI.escape(str)
+    end
+    
+    def uri_unescape( str )
+      URI.unescape(str)
+    end
+    
     # Matches two objects with their to_s values case insensitively
     # 
     # Example:

data/lib/mail/version.rb

@@ -3,7 +3,7 @@ module Mail
   module VERSION
     MAJOR = 2
     MINOR = 2
-    TINY  = 0
+    TINY  = 1
 
     STRING = [MAJOR, MINOR, TINY].join('.')
   end

data/lib/mail/version_specific/ruby_1_8.rb

@@ -21,6 +21,21 @@ module Mail
       '(' + str + ')'
     end
     
+    def Ruby18.escape_bracket( str )
+      re = /\\\>/
+      str = str.gsub(re) { |s| '>'}
+      re = /\\\</
+      str = str.gsub(re) { |s| '<'}
+      re = /([\<\>])/          # Only match unescaped parens
+      str.gsub(re) { |s| '\\' + s }
+    end
+    
+    def Ruby18.bracket( str )
+      str = $1 if str =~ /^\<(.*)?\>$/
+      str = escape_bracket( str )
+      '<' + str + '>'
+    end
+    
     def Ruby18.decode_base64(str)
       Base64.decode64(str)
     end

data/lib/mail/version_specific/ruby_1_9.rb

@@ -15,6 +15,17 @@ module Mail
       '(' + str + ')'
     end
     
+    def Ruby19.escape_bracket( str )
+      re = /(?<!\\)([\<\>])/          # Only match unescaped brackets
+      str.gsub(re) { |s| '\\' + s }
+    end
+
+    def Ruby19.bracket( str )
+      str = $1 if str =~ /^\<(.*)?\>$/
+      str = escape_bracket( str )
+      '<' + str + '>'
+    end
+
     def Ruby19.decode_base64(str)
       str.unpack( 'm' ).first.force_encoding(Encoding::BINARY)
     end

metadata

@@ -5,22 +5,22 @@ version: !ruby/object:Gem::Version
   segments: 
   - 2
   - 2
-  - 0
-  version: 2.2.0
+  - 1
+  version: 2.2.1
 platform: ruby
 authors: 
-- Mike Lindsaar
+- Mikel Lindsaar
 autorequire: 
 bindir: bin
 cert_chain: []
 
-date: 2010-04-11 00:00:00 +10:00
+date: 2010-05-13 00:00:00 +10:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
-  type: :runtime
   name: activesupport
-  version_requirements: &id001 !ruby/object:Gem::Requirement 
+  prerelease: false
+  requirement: &id001 !ruby/object:Gem::Requirement 
     requirements: 
     - - ">="
       - !ruby/object:Gem::Version 
@@ -29,24 +29,24 @@ dependencies:
         - 3
         - 4
         version: 2.3.4
-  requirement: *id001
-  prerelease: false
-- !ruby/object:Gem::Dependency 
   type: :runtime
+  version_requirements: *id001
+- !ruby/object:Gem::Dependency 
   name: mime-types
-  version_requirements: &id002 !ruby/object:Gem::Requirement 
+  prerelease: false
+  requirement: &id002 !ruby/object:Gem::Requirement 
     requirements: 
     - - ">="
       - !ruby/object:Gem::Version 
         segments: 
         - 0
         version: "0"
-  requirement: *id002
-  prerelease: false
-- !ruby/object:Gem::Dependency 
   type: :runtime
+  version_requirements: *id002
+- !ruby/object:Gem::Dependency 
   name: treetop
-  version_requirements: &id003 !ruby/object:Gem::Requirement 
+  prerelease: false
+  requirement: &id003 !ruby/object:Gem::Requirement 
     requirements: 
     - - ">="
       - !ruby/object:Gem::Version 
@@ -55,8 +55,8 @@ dependencies:
         - 4
         - 5
         version: 1.4.5
-  requirement: *id003
-  prerelease: false
+  type: :runtime
+  version_requirements: *id003
 description: A really Ruby Mail handler.
 email: raasdnil@gmail.com
 executables: []