email_reply_parser_ffcrm 0.5.0

data/LICENSE

@@ -0,0 +1,22 @@
+The MIT License
+
+Copyright (c) GitHub
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+

data/README.md

@@ -0,0 +1,103 @@
+# Email Reply Parser
+
+EmailReplyParser is a small library to parse plain text email content.
+See the rocco-documented source code for specifics on how it works.
+
+This is what GitHub uses to display comments that were created from
+email replies.  This code is being open sourced in an effort to
+crowdsource the quality of our email representation.
+
+See more at the [Rocco docs][rocco].
+
+[rocco]: http://help.github.com/code/email_reply_parser/
+
+## Problem?
+
+If you have a question about the behavior and formatting of email replies on GitHub, check out [support][support].  If you have a specific issue regarding this library, then hit up the [Issues][issues].
+
+[support]: http://support.github.com/
+[issues]: https://github.com/github/email_reply_parser/issues
+
+## Installation
+
+Get it from [GitHub][github] or `gem install email_reply_parser`.  Run `rake` to run the tests.
+
+[github]: https://github.com/github/email_reply_parser
+
+## Contribute
+
+If you'd like to hack on EmailReplyParser, start by forking the repo on GitHub:
+
+https://github.com/github/email_reply_parser
+
+The best way to get your changes merged back into core is as follows:
+
+* Clone down your fork
+* Create a thoughtfully named topic branch to contain your change
+* Hack away
+* Add tests and make sure everything still passes by running rake
+* If you are adding new functionality, document it in the README
+* Do not change the version number, I will do that on my end
+* If necessary, rebase your commits into logical chunks, without errors
+* Push the branch up to GitHub
+* Send a pull request to the `github/email_reply_parser` project.
+
+## Known Issues
+
+### Quoted Headers
+
+Quoted headers aren't picked up if there's an extra line break:
+
+    On <date>, <author> wrote:
+
+    > blah
+
+Also, they're not picked up if the email client breaks it up into
+multiple lines.  GMail breaks up any lines over 80 characters for you.
+
+    On <date>, <author>
+    wrote:
+    > blah
+
+Not to mention that we're search for "on" and "wrote".  It won't work
+with other languages.
+
+Possible solution: Remove "reply@reply.github.com" lines...
+
+### Weird Signatures
+
+Lines starting with `-` or `_` sometimes mark the beginning of
+signatures:
+
+    Hello
+
+    -- 
+    Rick
+
+Not everyone follows this convention:
+
+    Hello
+
+    Mr Rick Olson
+    Galactic President Superstar Mc Awesomeville
+    GitHub
+
+    **********************DISCLAIMER***********************************
+    * Note: blah blah blah                                            *
+    **********************DISCLAIMER***********************************
+
+
+
+### Strange Quoting
+
+Apparently, prefixing lines with `>` isn't universal either:
+
+    Hello
+
+    --
+    Rick
+
+    ________________________________________
+    From: Bob [reply@reply.github.com]
+    Sent: Monday, March 14, 2011 6:16 PM
+    To: Rick

data/Rakefile

@@ -0,0 +1,135 @@
+require 'rubygems'
+require 'rake'
+require 'date'
+
+#############################################################################
+#
+# Helper functions
+#
+#############################################################################
+
+def name
+  @name ||= "email_reply_parser"
+end
+
+def version
+  line = File.read("lib/#{name}.rb")[/^\s*VERSION\s*=\s*.*/]
+  line.match(/.*VERSION\s*=\s*['"](.*)['"]/)[1]
+end
+
+def date
+  Date.today.to_s
+end
+
+def rubyforge_project
+  name
+end
+
+def gemspec_file
+  "#{name}_ffcrm.gemspec"
+end
+
+def gem_file
+  "#{name}-#{version}.gem"
+end
+
+def replace_header(head, header_name)
+  head.sub!(/(\.#{header_name}\s*= ').*'/) { "#{$1}#{send(header_name)}'"}
+end
+
+#############################################################################
+#
+# Standard tasks
+#
+#############################################################################
+
+task :default => :test
+
+require 'rake/testtask'
+Rake::TestTask.new(:test) do |test|
+  test.libs << 'lib' << 'test'
+  test.pattern = 'test/**/*_test.rb'
+  test.verbose = true
+end
+
+desc "Open an irb session preloaded with this library"
+task :console do
+  sh "irb -rubygems -r ./lib/#{name}.rb"
+end
+
+#############################################################################
+#
+# Custom tasks (add your own tasks here)
+#
+#############################################################################
+
+
+
+#############################################################################
+#
+# Packaging tasks
+#
+#############################################################################
+
+desc "Create tag v#{version} and build and push #{gem_file} to Rubygems"
+task :release => :build do
+  unless `git branch` =~ /^\* master$/
+    puts "You must be on the master branch to release!"
+    exit!
+  end
+  sh "git commit --allow-empty -a -m 'Release #{version}'"
+  sh "git tag v#{version}"
+  sh "git push origin master"
+  sh "git push origin v#{version}"
+  sh "gem push pkg/#{name}-#{version}.gem"
+end
+
+desc "Build #{gem_file} into the pkg directory"
+task :build => :gemspec do
+  sh "mkdir -p pkg"
+  sh "gem build #{gemspec_file}"
+  sh "mv #{gem_file} pkg"
+end
+
+desc "Generate #{gemspec_file}"
+task :gemspec => :validate do
+  # read spec file and split out manifest section
+  spec = File.read(gemspec_file)
+  head, manifest, tail = spec.split("  # = MANIFEST =\n")
+
+  # replace name version and date
+  replace_header(head, :name)
+  replace_header(head, :version)
+  replace_header(head, :date)
+  #comment this out if your rubyforge_project has a different name
+  replace_header(head, :rubyforge_project)
+
+  # determine file list from git ls-files
+  files = `git ls-files`.
+    split("\n").
+    sort.
+    reject { |file| file =~ /^\./ }.
+    reject { |file| file =~ /^(rdoc|pkg)/ }.
+    map { |file| "    #{file}" }.
+    join("\n")
+
+  # piece file back together and write
+  manifest = "  s.files = %w[\n#{files}\n  ]\n"
+  spec = [head, manifest, tail].join("  # = MANIFEST =\n")
+  File.open(gemspec_file, 'w') { |io| io.write(spec) }
+  puts "Updated #{gemspec_file}"
+end
+
+desc "Validate #{gemspec_file}"
+task :validate do
+  libfiles = Dir['lib/*'] - ["lib/#{name}.rb", "lib/#{name}"]
+  unless libfiles.empty?
+    puts "Directory `lib` should only contain a `#{name}.rb` file and `#{name}` dir."
+    exit!
+  end
+  unless Dir['VERSION*'].empty?
+    puts "A `VERSION` file at root level violates Gem best practices."
+    exit!
+  end
+end
+

data/email_reply_parser_ffcrm.gemspec

@@ -0,0 +1,88 @@
+## This is the rakegem gemspec template. Make sure you read and understand
+## all of the comments. Some sections require modification, and others can
+## be deleted if you don't need them. Once you understand the contents of
+## this file, feel free to delete any comments that begin with two hash marks.
+## You can find comprehensive Gem::Specification documentation, at
+## http://docs.rubygems.org/read/chapter/20
+Gem::Specification.new do |s|
+  s.specification_version = 2 if s.respond_to? :specification_version=
+  s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
+  s.rubygems_version = '1.3.5'
+
+  ## Leave these as is they will be modified for you by the rake gemspec task.
+  ## If your rubyforge_project name is different, then edit it and comment out
+  ## the sub! line in the Rakefile
+  s.name              = 'email_reply_parser_ffcrm'
+  s.version           = '0.5.0'
+  s.date              = '2012-05-09'
+
+  ## Make sure your summary is short. The description may be as long
+  ## as you like.
+  s.summary     = "Short description used in Gem listings."
+  s.description = "Long description. Maybe copied from the README."
+
+  ## List the primary authors. If there are a bunch of authors, it's probably
+  ## better to set the email to an email list or something. If you don't have
+  ## a custom homepage, consider using your GitHub URL or the like.
+  s.authors  = ["Rick Olson"]
+  s.email    = 'technoweenie@gmail.com'
+  s.homepage = 'http://github.com/github/email_reply_parser'
+
+  ## This gets added to the $LOAD_PATH so that 'lib/NAME.rb' can be required as
+  ## require 'NAME.rb' or'/lib/NAME/file.rb' can be as require 'NAME/file.rb'
+  s.require_paths = %w[lib]
+
+  ## This sections is only necessary if you have C extensions.
+  #s.require_paths << 'ext'
+  #s.extensions = %w[ext/extconf.rb]
+
+  ## If your gem includes any executables, list them here.
+  #s.executables = ["name"]
+  #s.default_executable = 'name'
+
+  ## Specify any RDoc options here. You'll want to add your README and
+  ## LICENSE files to the extra_rdoc_files list.
+  s.rdoc_options = ["--charset=UTF-8"]
+  s.extra_rdoc_files = %w[README.md LICENSE]
+
+  ## List your runtime dependencies here. Runtime dependencies are those
+  ## that are needed for an end user to actually USE your code.
+  #s.add_dependency('DEPNAME', [">= 1.1.0", "< 2.0.0"])
+
+  ## List your development dependencies here. Development dependencies are
+  ## those that are only needed during development
+  #s.add_development_dependency('DEVDEPNAME', [">= 1.1.0", "< 2.0.0"])
+
+  ## Leave this section as-is. It will be automatically generated from the
+  ## contents of your Git repository via the gemspec task. DO NOT REMOVE
+  ## THE MANIFEST COMMENTS, they are used as delimiters by the task.
+  # = MANIFEST =
+  s.files = %w[
+    LICENSE
+    README.md
+    Rakefile
+    email_reply_parser_ffcrm.gemspec
+    lib/email_reply_parser.rb
+    test/email_reply_parser_test.rb
+    test/emails/correct_sig.txt
+    test/emails/email_1_1.txt
+    test/emails/email_1_2.txt
+    test/emails/email_1_3.txt
+    test/emails/email_1_4.txt
+    test/emails/email_1_5.txt
+    test/emails/email_1_6.txt
+    test/emails/email_2_1.txt
+    test/emails/email_2_2.txt
+    test/emails/email_BlackBerry.txt
+    test/emails/email_bullets.txt
+    test/emails/email_iPhone.txt
+    test/emails/email_multi_word_sent_from_my_mobile_device.txt
+    test/emails/email_sent_from_my_not_signature.txt
+  ]
+  # = MANIFEST =
+
+  ## Test files will be grabbed from the file list. Make sure the path glob
+  ## matches what you actually use.
+  s.test_files = s.files.select { |path| path =~ /^test\/.*_test\.rb/ }
+end
+

data/lib/email_reply_parser.rb

@@ -0,0 +1,265 @@
+require 'strscan'
+
+# EmailReplyParser is a small library to parse plain text email content.  The
+# goal is to identify which fragments are quoted, part of a signature, or
+# original body content.  We want to support both top and bottom posters, so
+# no simple "REPLY ABOVE HERE" content is used.
+#
+# Beyond RFC 5322 (which is handled by the [Ruby mail gem][mail]), there aren't
+# any real standards for how emails are created.  This attempts to parse out
+# common conventions for things like replies:
+#
+#     this is some text
+#
+#     On <date>, <author> wrote:
+#     > blah blah
+#     > blah blah
+#
+# ... and signatures:
+#
+#     this is some text
+#
+#     --
+#     Bob
+#     http://homepage.com/~bob
+#
+# Each of these are parsed into Fragment objects.
+#
+# EmailReplyParser also attempts to figure out which of these blocks should
+# be hidden from users.
+#
+# [mail]: https://github.com/mikel/mail
+class EmailReplyParser
+  VERSION = "0.5.0"
+
+  # Public: Splits an email body into a list of Fragments.
+  #
+  # text - A String email body.
+  #
+  # Returns an Email instance.
+  def self.read(text)
+    Email.new.read(text)
+  end
+
+  # Public: Get the text of the visible portions of the given email body.
+  #
+  # text - A String email body.
+  #
+  # Returns a String.
+  def self.parse_reply(text)
+    self.read(text).visible_text
+  end
+
+  ### Emails
+
+  # An Email instance represents a parsed body String.
+  class Email
+    # Emails have an Array of Fragments.
+    attr_reader :fragments
+
+    def initialize
+      @fragments = []
+    end
+
+    # Public: Gets the combined text of the visible fragments of the email body.
+    #
+    # Returns a String.
+    def visible_text
+      fragments.select{|f| !f.hidden?}.map{|f| f.to_s}.join("\n").rstrip
+    end
+
+    # Splits the given text into a list of Fragments.  This is roughly done by
+    # reversing the text and parsing from the bottom to the top.  This way we
+    # can check for 'On <date>, <author> wrote:' lines above quoted blocks.
+    #
+    # text - A String email body.
+    #
+    # Returns this same Email instance.
+    def read(text)
+      # Check for multi-line reply headers. Some clients break up
+      # the "On DATE, NAME <EMAIL> wrote:" line into multiple lines.
+      if text =~ /^(On(.+)wrote:)$/m
+        # Remove all new lines from the reply header.
+        text.gsub! $1, $1.gsub("\n", " ")
+      end
+
+      # Some users may reply directly above a line of underscores.
+      # In order to ensure that these fragments are split correctly,
+      # make sure that all lines of underscores are preceded by
+      # at least two newline characters.
+      text.gsub!(/([^\n])(?=\n_{7}_+)$/m, "\\1\n")
+
+      # The text is reversed initially due to the way we check for hidden
+      # fragments.
+      text = text.reverse
+
+      # This determines if any 'visible' Fragment has been found.  Once any
+      # visible Fragment is found, stop looking for hidden ones.
+      @found_visible = false
+
+      # This instance variable points to the current Fragment.  If the matched
+      # line fits, it should be added to this Fragment.  Otherwise, finish it
+      # and start a new Fragment.
+      @fragment = nil
+
+      # Use the StringScanner to pull out each line of the email content.
+      @scanner = StringScanner.new(text)
+      while line = @scanner.scan_until(/\n/)
+        scan_line(line)
+      end
+
+      # Be sure to parse the last line of the email.
+      if (last_line = @scanner.rest.to_s).size > 0
+        scan_line(last_line)
+      end
+
+      # Finish up the final fragment.  Finishing a fragment will detect any
+      # attributes (hidden, signature, reply), and join each line into a
+      # string.
+      finish_fragment
+
+      @scanner = @fragment = nil
+
+      # Now that parsing is done, reverse the order.
+      @fragments.reverse!
+      self
+    end
+
+  private
+    EMPTY = "".freeze
+    SIG_REGEX = /(--|__|\w-$)|(^(\w+\s*){1,3} #{"Sent from my".reverse}$)/
+    
+    ### Line-by-Line Parsing
+
+    # Scans the given line of text and figures out which fragment it belongs
+    # to.
+    #
+    # line - A String line of text from the email.
+    #
+    # Returns nothing.
+    def scan_line(line)
+      line.chomp!("\n")
+      line.lstrip! unless line =~ SIG_REGEX
+
+      # We're looking for leading `>`'s to see if this line is part of a
+      # quoted Fragment.
+      is_quoted = !!(line =~ /(>+)$/)
+
+      # Mark the current Fragment as a signature if the current line is empty
+      # and the Fragment starts with a common signature indicator.
+      if @fragment && line == EMPTY
+        if @fragment.lines.last =~ SIG_REGEX
+          @fragment.signature = true
+          finish_fragment
+        end
+      end
+
+      # If the line matches the current fragment, add it.  Note that a common
+      # reply header also counts as part of the quoted Fragment, even though
+      # it doesn't start with `>`.
+      if @fragment &&
+          ((@fragment.quoted? == is_quoted) ||
+           (@fragment.quoted? && (quote_header?(line) || line == EMPTY)))
+        @fragment.lines << line
+
+      # Otherwise, finish the fragment and start a new one.
+      else
+        finish_fragment
+        @fragment = Fragment.new(is_quoted, line)
+      end
+    end
+
+    # Detects if a given line is a header above a quoted area.  It is only
+    # checked for lines preceding quoted regions.
+    #
+    # line - A String line of text from the email.
+    #
+    # Returns true if the line is a valid header, or false.
+    def quote_header?(line)
+      line =~ /^:etorw.*nO$/
+    end
+
+    # Builds the fragment string and reverses it, after all lines have been
+    # added.  It also checks to see if this Fragment is hidden.  The hidden
+    # Fragment check reads from the bottom to the top.
+    #
+    # Any quoted Fragments or signature Fragments are marked hidden if they
+    # are below any visible Fragments.  Visible Fragments are expected to
+    # contain original content by the author.  If they are below a quoted
+    # Fragment, then the Fragment should be visible to give context to the
+    # reply.
+    #
+    #     some original text (visible)
+    #
+    #     > do you have any two's? (quoted, visible)
+    #
+    #     Go fish! (visible)
+    #
+    #     > --
+    #     > Player 1 (quoted, hidden)
+    #
+    #     --
+    #     Player 2 (signature, hidden)
+    #
+    def finish_fragment
+      if @fragment
+        @fragment.finish
+        if !@found_visible
+          if @fragment.quoted? || @fragment.signature? ||
+              @fragment.to_s.strip == EMPTY
+            @fragment.hidden = true
+          else
+            @found_visible = true
+          end
+        end
+        @fragments << @fragment
+      end
+      @fragment = nil
+    end
+  end
+
+  ### Fragments
+
+  # Represents a group of paragraphs in the email sharing common attributes.
+  # Paragraphs should get their own fragment if they are a quoted area or a
+  # signature.
+  class Fragment < Struct.new(:quoted, :signature, :hidden)
+    # This is an Array of String lines of content.  Since the content is
+    # reversed, this array is backwards, and contains reversed strings.
+    attr_reader :lines,
+
+    # This is reserved for the joined String that is build when this Fragment
+    # is finished.
+      :content
+
+    def initialize(quoted, first_line)
+      self.signature = self.hidden = false
+      self.quoted = quoted
+      @lines      = [first_line]
+      @content    = nil
+      @lines.compact!
+    end
+
+    alias quoted?    quoted
+    alias signature? signature
+    alias hidden?    hidden
+
+    # Builds the string content by joining the lines and reversing them.
+    #
+    # Returns nothing.
+    def finish
+      @content = @lines.join("\n")
+      @lines = nil
+      @content.reverse!
+    end
+
+    def to_s
+      @content
+    end
+
+    def inspect
+      to_s.inspect
+    end
+  end
+end
+

data/test/email_reply_parser_test.rb

@@ -0,0 +1,154 @@
+require 'rubygems'
+require 'test/unit'
+require 'pathname'
+require 'pp'
+
+dir = Pathname.new File.expand_path(File.dirname(__FILE__))
+require dir + '..' + 'lib' + 'email_reply_parser'
+
+EMAIL_FIXTURE_PATH = dir + 'emails'
+
+class EmailReplyParserTest < Test::Unit::TestCase
+  def test_reads_simple_body
+    reply = email(:email_1_1)
+    assert_equal 3, reply.fragments.size
+
+    assert reply.fragments.none? { |f| f.quoted? }
+    assert_equal [false, true, true],
+      reply.fragments.map { |f| f.signature? }
+    assert_equal [false, true, true],
+      reply.fragments.map { |f| f.hidden? }
+
+    assert_equal "Hi folks
+
+What is the best way to clear a Riak bucket of all key, values after
+running a test?
+I am currently using the Java HTTP API.\n", reply.fragments[0].to_s
+
+    assert_equal "-Abhishek Kona\n\n", reply.fragments[1].to_s
+  end
+
+  def test_reads_top_post
+    reply = email(:email_1_3)
+    assert_equal 5, reply.fragments.size
+
+    assert_equal [false, false, true, false, false],
+      reply.fragments.map { |f| f.quoted? }
+    assert_equal [false, true, true, true, true],
+      reply.fragments.map { |f| f.hidden? }
+    assert_equal [false, true, false, false, true],
+      reply.fragments.map { |f| f.signature? }
+
+    assert_match /^Oh thanks.\n\nHaving/, reply.fragments[0].to_s
+    assert_match /^-A/, reply.fragments[1].to_s
+    assert_match /^On [^\:]+\:/, reply.fragments[2].to_s
+    assert_match /^_/, reply.fragments[4].to_s
+  end
+
+  def test_reads_bottom_post
+    reply = email(:email_1_2)
+    assert_equal 6, reply.fragments.size
+
+    assert_equal [false, true, false, true, false, false],
+      reply.fragments.map { |f| f.quoted? }
+    assert_equal [false, false, false, false, false, true],
+      reply.fragments.map { |f| f.signature? }
+    assert_equal [false, false, false, true, true, true],
+      reply.fragments.map { |f| f.hidden? }
+
+    assert_equal "Hi,", reply.fragments[0].to_s
+    assert_match /^On [^\:]+\:/, reply.fragments[1].to_s
+    assert_match /^You can list/, reply.fragments[2].to_s
+    assert_match /^> /, reply.fragments[3].to_s
+    assert_match /^_/, reply.fragments[5].to_s
+  end
+
+  def test_recognizes_date_string_above_quote
+    reply = email :email_1_4
+
+    assert_match /^Awesome/, reply.fragments[0].to_s
+    assert_match /^On/,      reply.fragments[1].to_s
+    assert_match /Loader/,   reply.fragments[1].to_s
+  end
+
+  def test_a_complex_body_with_only_one_fragment
+    reply = email :email_1_5
+
+    assert_equal 1, reply.fragments.size
+  end
+
+  def test_reads_email_with_correct_signature
+    reply = email :correct_sig
+    
+    assert_equal 2, reply.fragments.size
+    assert_equal [false, false], reply.fragments.map { |f| f.quoted? }
+    assert_equal [false, true], reply.fragments.map { |f| f.signature? }
+    assert_equal [false, true], reply.fragments.map { |f| f.hidden? }
+    assert_match /^-- \nrick/, reply.fragments[1].to_s
+  end
+
+  def test_deals_with_multiline_reply_headers
+    reply = email :email_1_6
+
+    assert_match /^I get/,   reply.fragments[0].to_s
+    assert_match /^On/,      reply.fragments[1].to_s
+    assert_match /Was this/, reply.fragments[1].to_s
+  end
+
+  def test_does_not_modify_input_string
+    original = "The Quick Brown Fox Jumps Over The Lazy Dog"
+    EmailReplyParser.read original
+    assert_equal "The Quick Brown Fox Jumps Over The Lazy Dog", original
+  end
+
+  def test_returns_only_the_visible_fragments_as_a_string
+    reply = email(:email_2_1)
+    assert_equal reply.fragments.select{|r| !r.hidden?}.map{|r| r.to_s}.join("\n").rstrip, reply.visible_text
+  end
+
+  def test_parse_out_just_top_for_outlook_reply
+    body = IO.read EMAIL_FIXTURE_PATH.join("email_2_1.txt").to_s
+    assert_equal "Outlook with a reply", EmailReplyParser.parse_reply(body)
+  end
+
+  def test_parse_out_just_top_for_outlook_with_reply_directly_above_line
+    body = IO.read EMAIL_FIXTURE_PATH.join("email_2_2.txt").to_s
+    assert_equal "Outlook with a reply directly above line", EmailReplyParser.parse_reply(body)
+  end
+
+  def test_parse_out_sent_from_iPhone
+    body = IO.read EMAIL_FIXTURE_PATH.join("email_iPhone.txt").to_s
+    assert_equal "Here is another email", EmailReplyParser.parse_reply(body)
+  end
+
+  def test_parse_out_sent_from_BlackBerry
+    body = IO.read EMAIL_FIXTURE_PATH.join("email_BlackBerry.txt").to_s
+    assert_equal "Here is another email", EmailReplyParser.parse_reply(body)
+  end
+
+  def test_parse_out_send_from_multiword_mobile_device
+    body = IO.read EMAIL_FIXTURE_PATH.join("email_multi_word_sent_from_my_mobile_device.txt").to_s
+    assert_equal "Here is another email", EmailReplyParser.parse_reply(body)
+  end
+
+  def test_do_not_parse_out_send_from_in_regular_sentence
+    body = IO.read EMAIL_FIXTURE_PATH.join("email_sent_from_my_not_signature.txt").to_s
+    assert_equal "Here is another email\n\nSent from my desk, is much easier then my mobile phone.", EmailReplyParser.parse_reply(body)
+  end
+
+  def test_retains_bullets
+    body = IO.read EMAIL_FIXTURE_PATH.join("email_bullets.txt").to_s
+    assert_equal "test 2 this should list second\n\nand have spaces\n\nand retain this formatting\n\n\n   - how about bullets\n   - and another", 
+      EmailReplyParser.parse_reply(body)
+      end
+
+  def test_parse_reply
+    body = IO.read EMAIL_FIXTURE_PATH.join("email_1_2.txt").to_s
+    assert_equal EmailReplyParser.read(body).visible_text, EmailReplyParser.parse_reply(body)
+  end
+
+  def email(name)
+    body = IO.read EMAIL_FIXTURE_PATH.join("#{name}.txt").to_s
+    EmailReplyParser.read body
+  end
+end

data/test/emails/correct_sig.txt

@@ -0,0 +1,4 @@
+this is an email with a correct -- signature.
+
+-- 
+rick

data/test/emails/email_1_1.txt

@@ -0,0 +1,13 @@
+Hi folks
+
+What is the best way to clear a Riak bucket of all key, values after 
+running a test?
+I am currently using the Java HTTP API.
+
+-Abhishek Kona
+
+
+_______________________________________________
+riak-users mailing list
+riak-users@lists.basho.com
+http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

data/test/emails/email_1_2.txt

@@ -0,0 +1,51 @@
+Hi,
+On Tue, 2011-03-01 at 18:02 +0530, Abhishek Kona wrote:
+> Hi folks
+> 
+> What is the best way to clear a Riak bucket of all key, values after 
+> running a test?
+> I am currently using the Java HTTP API.
+
+You can list the keys for the bucket and call delete for each. Or if you
+put the keys (and kept track of them in your test) you can delete them
+one at a time (without incurring the cost of calling list first.)
+
+Something like:
+
+        String bucket = "my_bucket";
+        BucketResponse bucketResponse = riakClient.listBucket(bucket);
+        RiakBucketInfo bucketInfo = bucketResponse.getBucketInfo();
+        
+        for(String key : bucketInfo.getKeys()) {
+            riakClient.delete(bucket, key);
+        }
+
+
+would do it.
+
+See also 
+
+http://wiki.basho.com/REST-API.html#Bucket-operations
+
+which says 
+
+"At the moment there is no straightforward way to delete an entire
+Bucket. There is, however, an open ticket for the feature. To delete all
+the keys in a bucket, you’ll need to delete them all individually."
+
+> 
+> -Abhishek Kona
+> 
+> 
+> _______________________________________________
+> riak-users mailing list
+> riak-users@lists.basho.com
+> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
+
+
+
+
+_______________________________________________
+riak-users mailing list
+riak-users@lists.basho.com
+http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

data/test/emails/email_1_3.txt

@@ -0,0 +1,55 @@
+Oh thanks.
+
+Having the function would be great.
+
+-Abhishek Kona
+
+On 01/03/11 7:07 PM, Russell Brown wrote:
+> Hi,
+> On Tue, 2011-03-01 at 18:02 +0530, Abhishek Kona wrote:
+>> Hi folks
+>>
+>> What is the best way to clear a Riak bucket of all key, values after
+>> running a test?
+>> I am currently using the Java HTTP API.
+> You can list the keys for the bucket and call delete for each. Or if you
+> put the keys (and kept track of them in your test) you can delete them
+> one at a time (without incurring the cost of calling list first.)
+>
+> Something like:
+>
+>          String bucket = "my_bucket";
+>          BucketResponse bucketResponse = riakClient.listBucket(bucket);
+>          RiakBucketInfo bucketInfo = bucketResponse.getBucketInfo();
+>
+>          for(String key : bucketInfo.getKeys()) {
+>              riakClient.delete(bucket, key);
+>          }
+>
+>
+> would do it.
+>
+> See also
+>
+> http://wiki.basho.com/REST-API.html#Bucket-operations
+>
+> which says
+>
+> "At the moment there is no straightforward way to delete an entire
+> Bucket. There is, however, an open ticket for the feature. To delete all
+> the keys in a bucket, you’ll need to delete them all individually."
+>
+>> -Abhishek Kona
+>>
+>>
+>> _______________________________________________
+>> riak-users mailing list
+>> riak-users@lists.basho.com
+>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
+>
+
+
+_______________________________________________
+riak-users mailing list
+riak-users@lists.basho.com
+http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

data/test/emails/email_1_4.txt

@@ -0,0 +1,5 @@
+Awesome! I haven't had another problem with it.
+
+On Aug 22, 2011, at 7:37 PM, defunkt<reply@reply.github.com> wrote:
+
+> Loader seems to be working well.

data/test/emails/email_1_5.txt

@@ -0,0 +1,15 @@
+One: Here's what I've got.
+
+- This would be the first bullet point that wraps to the second line
+to the next
+- This is the second bullet point and it doesn't wrap
+- This is the third bullet point and I'm having trouble coming up with enough
+to say
+- This is the fourth bullet point
+
+Two:
+- Here is another bullet point
+- And another one
+
+This is a paragraph that talks about a bunch of stuff. It goes on and on
+for a while.

data/test/emails/email_1_6.txt

@@ -0,0 +1,15 @@
+I get proper rendering as well.
+
+Sent from a magnificent torch of pixels
+
+On Dec 16, 2011, at 12:47 PM, Corey Donohoe
+<reply@reply.github.com>
+wrote:
+
+> Was this caching related or fixed already?  I get proper rendering here.
+>
+> ![](https://img.skitch.com/20111216-m9munqjsy112yqap5cjee5wr6c.jpg)
+>
+> ---
+> Reply to this email directly or view it on GitHub:
+> https://github.com/github/github/issues/2278#issuecomment-3182418

data/test/emails/email_2_1.txt

@@ -0,0 +1,25 @@
+Outlook with a reply
+
+
+ ------------------------------
+
+*From:* Google Apps Sync Team [mailto:mail-noreply@google.com]
+*Sent:* Thursday, February 09, 2012 1:36 PM
+*To:* jow@xxxx.com
+*Subject:* Google Apps Sync was updated!
+
+
+
+Dear Google Apps Sync user,
+
+Google Apps Sync for Microsoft Outlook® was recently updated. Your computer
+now has the latest version (version 2.5). This release includes bug fixes
+to improve product reliability. For more information about these and other
+changes, please see the help article here:
+
+http://www.google.com/support/a/bin/answer.py?answer=153463
+
+Sincerely,
+
+The Google Apps Sync Team.
+

data/test/emails/email_2_2.txt

@@ -0,0 +1,10 @@
+Outlook with a reply directly above line
+________________________________________
+From: CRM Comments [crm-comment@example.com]
+Sent: Friday, 23 March 2012 5:08 p.m.
+To: John S. Greene
+Subject: [contact:106] John Greene
+
+A new comment has been added to the Contact named 'John Greene':
+
+I am replying to a comment.

data/test/emails/email_BlackBerry.txt

@@ -0,0 +1,3 @@
+Here is another email
+
+Sent from my BlackBerry

data/test/emails/email_bullets.txt

@@ -0,0 +1,22 @@
+test 2 this should list second
+
+and have spaces
+
+and retain this formatting
+
+
+   - how about bullets
+   - and another
+
+
+On Fri, Feb 24, 2012 at 10:19 AM, <examples@email.goalengine.com> wrote:
+
+> Give us an example of how you applied what they learned to achieve
+> something in your organization
+
+
+
+
+-- 
+
+*Joe Smith | Director, Product Management*

data/test/emails/email_iPhone.txt

@@ -0,0 +1,3 @@
+Here is another email
+
+Sent from my iPhone

data/test/emails/email_multi_word_sent_from_my_mobile_device.txt

@@ -0,0 +1,3 @@
+Here is another email
+
+Sent from my Verizon Wireless BlackBerry

data/test/emails/email_sent_from_my_not_signature.txt

@@ -0,0 +1,3 @@
+Here is another email
+
+Sent from my desk, is much easier then my mobile phone.

metadata

@@ -0,0 +1,68 @@
+--- !ruby/object:Gem::Specification
+name: email_reply_parser_ffcrm
+version: !ruby/object:Gem::Version
+  version: 0.5.0
+  prerelease: 
+platform: ruby
+authors:
+- Rick Olson
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2012-05-09 00:00:00.000000000 Z
+dependencies: []
+description: Long description. Maybe copied from the README.
+email: technoweenie@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files:
+- README.md
+- LICENSE
+files:
+- LICENSE
+- README.md
+- Rakefile
+- email_reply_parser_ffcrm.gemspec
+- lib/email_reply_parser.rb
+- test/email_reply_parser_test.rb
+- test/emails/correct_sig.txt
+- test/emails/email_1_1.txt
+- test/emails/email_1_2.txt
+- test/emails/email_1_3.txt
+- test/emails/email_1_4.txt
+- test/emails/email_1_5.txt
+- test/emails/email_1_6.txt
+- test/emails/email_2_1.txt
+- test/emails/email_2_2.txt
+- test/emails/email_BlackBerry.txt
+- test/emails/email_bullets.txt
+- test/emails/email_iPhone.txt
+- test/emails/email_multi_word_sent_from_my_mobile_device.txt
+- test/emails/email_sent_from_my_not_signature.txt
+homepage: http://github.com/github/email_reply_parser
+licenses: []
+post_install_message: 
+rdoc_options:
+- --charset=UTF-8
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.24
+signing_key: 
+specification_version: 2
+summary: Short description used in Gem listings.
+test_files:
+- test/email_reply_parser_test.rb