extended_email_reply_parser 0.1.0 → 0.2.0

data/CHANGELOG.md

@@ -10,6 +10,28 @@ This project adheres to [Semantic Versioning](http://semver.org/).
 ### Removed
 ### Fixed
 
+## ExtendedEmailReplyParser 0.2.0 (2016-07-22)
+### Added
+- `Parsers::Base#hide_everything_after(expressions)` is useful when email clients do not quote the previous conversation. This parser method hides everything lead by a series of expressions, e.g. `hide_everything_after %w(From: Sent: To:)`.
+- `Parsers::Base#except_in_visible_block_quotes`. Within this block, `hide_everything_after` is not applied. This is useful when a quote is already marked as to be shown.
+- German parser `Parsers::I18nDe`, which removes previous conversation by searching for the phrases "Gesendet: Von: An:" and "Am ... schrieb ...:".
+- Support for i18n-ed header lines. The github parser only knows "On ... wrote". Since this is needed when the github parser runs, specify additional regexes in the class header of the parsers using `add_quote_header_regex`, for example: `add_quote_header_regex '^Am .* schrieb.*$'`.
+- The German parser adds the regex for quote headers like "Am ... schrieb ...:".
+- Remove empty lines between quote lines:
+
+        > Hi,
+        > how are you doing?
+        > Cheers
+
+  rather than
+
+        > Hi,
+
+        > how are you doing?
+
+        > Cheers
+- English parser `Parsers::I18nEn`, which removes previous conversation by searching for the phrases "From: Sent: To".
+
 ## ExtendedEmailReplyParser 0.1.0 (2016-07-22)
 ### Added
 - `ExtendedEmailReplyParser.read "/path/to/email.eml"` returns the corresponding `Mail::Message` object.

data/Gemfile

@@ -2,3 +2,5 @@ source 'https://rubygems.org'
 
 # Specify your gem's dependencies in extended_email_reply_parser.gemspec
 gemspec
+
+gem "codeclimate-test-reporter", group: :test, require: nil

data/README.md

@@ -1,6 +1,6 @@
 # ExtendedEmailReplyParser
 
-[![Build Status](https://travis-ci.org/fiedl/extended_email_reply_parser.svg?branch=master)](https://travis-ci.org/fiedl/extended_email_reply_parser)
+[![Join the chat at https://gitter.im/fiedl/extended_email_reply_parser](https://badges.gitter.im/fiedl/extended_email_reply_parser.svg)](https://gitter.im/fiedl/extended_email_reply_parser?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) [![Build Status](https://travis-ci.org/fiedl/extended_email_reply_parser.svg?branch=master)](https://travis-ci.org/fiedl/extended_email_reply_parser) [![Code Climate](https://codeclimate.com/github/fiedl/extended_email_reply_parser/badges/gpa.svg)](https://codeclimate.com/github/fiedl/extended_email_reply_parser) [![Test Coverage](https://codeclimate.com/github/fiedl/extended_email_reply_parser/badges/coverage.svg)](https://codeclimate.com/github/fiedl/extended_email_reply_parser/coverage) [![Gem Version](https://badge.fury.io/rb/extended_email_reply_parser.svg)](https://badge.fury.io/rb/extended_email_reply_parser) [![Documentation](https://img.shields.io/badge/documentation-rubydoc.info-blue.svg)](http://www.rubydoc.info/github/fiedl/extended_email_reply_parser/)
 
 When implementing a "reply or comment by email" feature, it's neccessary to filter out signatures and the previous conversation. One needs to extract just the relevant parts for the conversation or comment section of the application. This is what this [ruby](https://www.ruby-lang.org) gem helps to do.
 
@@ -81,7 +81,6 @@ EmailParsers::ShoutParser.parse \
 ```
 
 
-
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -105,6 +104,108 @@ After checking out the repo, run `bin/setup` to install dependencies. Then, run
 
 To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
 
+### Helper methods for writing parsers
+
+To accomplish the most common parsing operations, there are a couple of helper methods.
+
+This, for example, is the [English parser](lib/extended_email_reply_parser/parsers/i18n_en.rb).
+
+```ruby
+module ExtendedEmailReplyParser
+  class Parsers::I18nEn < Parsers::Base
+
+    def parse
+      except_in_visible_block_quotes do
+        hide_everything_after ["From: ", "Sent: ", "To: "]
+      end
+    end
+
+  end
+end
+```
+
+#### `add_quote_header_regex`
+
+The [github parser](https://github.com/github/email_reply_parser) needs to know how to identify the header line of quotes, for example "On Tue, 2011-03-01 at 18:02 +0530, Abhishek Kona wrote":
+
+    Hi,
+
+    On Tue, 2011-03-01 at 18:02 +0530, Abhishek Kona wrote:
+    > Hi folks
+    >
+    > What is the best way to clear a Riak bucket of all key, values after
+    > running a test?
+    > I am currently using the Java HTTP API.
+
+    You can list the keys for the bucket and call delete for each. Or if you
+    put the keys (and kept track of them in your test) you can delete them
+    one at a time (without incurring the cost of calling list first.)
+
+By default, it uses the regex `/^On .* wrote:$/` for that. To make it recognize other header lines, specify their patterns using `add_quote_header_regex`.
+
+Since this is needed by the github parser, i.e. possibly before the `parse` method of your custom parser is run, make sure to add the quote header regex in the class head:
+
+```ruby
+module ExtendedEmailReplyParser
+  class Parsers::I18nDe < Parsers::Base
+    add_quote_header_regex '^Am .* schrieb.*$'
+    # ...
+  end
+end
+```
+
+#### `hide_everything_after`
+
+Some email clients do not quote the previous conversation.
+
+    Hi Chris,
+    this is great, thanks!
+    Cheers, John
+
+
+    From: Chris <chris@example.com>
+    Sent: Saturday, July 09, 2016 3:27 PM
+    To: John <john@example.com>
+    Subject: The solution!
+
+    Hi John,
+    I've just found a solution to our big problem!
+    ...
+
+To remove the previous conversation, tell the parser expressions to identify where start of the previous conversation:
+
+```ruby
+module ExtendedEmailReplyParser
+  class Parsers::I18nEn < Parsers::Base
+    def parse
+      except_in_visible_block_quotes do
+        hide_everything_after ["From: ", "Sent: ", "To: "]
+      end
+      # ...
+    end
+  end
+end
+```
+
+(The parser will combine the expressions to a regex: `/(#{expressions.join(".*?")}.*?\n)/m`, for example: `/(From: .*?Sent: .*?To: .*?\n)/m`.)
+
+To avoid cutting off the email within a visible quote, wrap the `hide_everything_after` within a `except_in_visible_block_quotes` block as shown above.
+
+    Hi Chris,
+
+    > From: Chris <chris@example.com>
+    > Sent: Saturday, July 09, 2016 3:27 PM
+    > To: John <john@example.com>
+    > Subject: The solution!
+    >
+    > Hi John,
+    > I've just found a solution to our big problem!
+
+    this is great, thanks!
+    Cheers, John
+
+If not wrapped in `except_in_visible_block_quotes`, the parsed email would just be "Hi Chris,", because everything after "From: Sent: To:" would be cut off.
+
 ## Contributing
 
 Bug reports and pull requests are welcome on GitHub at https://github.com/fiedl/extended_email_reply_parser.

data/lib/extended_email_reply_parser/email_reply_parser/email.rb

@@ -0,0 +1,68 @@
+class EmailReplyParser
+  class Email
+    def hide_everything_after(expressions)
+      split_regex = /(#{expressions.join(".*?")}.*?\n)/m
+      split_fragments_at split_regex
+    end
+
+    def remove_empty_lines_between_block_quote_lines
+      @fragments = @fragments.collect do |fragment|
+        if fragment.quoted?
+          fragment.content = fragment.content.gsub /\n *?\n>/m, "\n>"
+        end
+        fragment
+      end
+    end
+
+    def split_fragments_at(regex)
+      @fragments = @fragments.collect do |fragment|
+        if fragment.to_s
+          first_text, *rest = fragment.to_s.split(regex)
+
+          first_fragment = Fragment.new(false, first_text)
+          first_fragment.quoted = fragment.quoted
+          first_fragment.hidden = fragment.hidden
+          first_fragment.signature = fragment.signature
+          first_fragment.content = first_text
+
+          hidden_fragment = Fragment.new(true, rest.join("\n"))
+          hidden_fragment.content = rest.join("\n")
+
+          hidden_fragment.quoted = true
+          if @except_in_visible_block_quotes
+            hidden_fragment.hidden = true unless fragment.quoted? and not fragment.hidden?
+          else
+            hidden_fragment.hidden = true
+          end
+
+          [first_fragment, hidden_fragment]
+        end
+      end.flatten - [nil]
+      @fragments = @fragments.select { |fragment| fragment.to_s && fragment.to_s != "" }
+    end
+
+    def except_in_visible_block_quotes
+      @except_in_visible_block_quotes = true
+      yield
+      @except_in_visible_block_quotes = false
+    end
+
+    private
+
+    # Detects if a given line is a header above a quoted area.  It is only
+    # checked for lines preceding quoted regions.
+    #
+    # line - A String line of text from the email.
+    #
+    # Returns true if the line is a valid header, or false.
+    #
+    # This method overrides the original in order to include the different
+    # regex defined in the different ExtendedEmailReplyParser::Parsers.
+    #
+    def quote_header?(line)
+      regex = ExtendedEmailReplyParser::Parsers::Base.quote_header_regexes.join("|")
+      line.reverse =~ /#{regex}/
+    end
+
+  end
+end

data/lib/extended_email_reply_parser/email_reply_parser/fragment.rb

@@ -0,0 +1,9 @@
+class EmailReplyParser
+  class Fragment
+
+    def content=(new_content)
+      @content = new_content
+    end
+
+  end
+end

data/lib/extended_email_reply_parser/parsers/base.rb

@@ -1,16 +1,172 @@
 module ExtendedEmailReplyParser
   class Parsers::Base
+    @@quote_header_regexes ||= []
 
     attr_accessor :text
 
     def initialize(text_before_parsing)
       self.text = text_before_parsing
+
+      # The `EmailReplyParser::Email` is extended in this gem.
+      # Have a look at:
+      #
+      #   lib/extended_email_reply_parser/email_reply_parser/email.rb
+      #
+      @email = EmailReplyParser::Email.new.read(text)
     end
 
+    # This `parse` method of the `Parsers::Base` will be overridden
+    # by the individual parsers.
+    #
+    # The text before parsing is accessed with `text`.
+    # The method `parse` is expected to return the parsed text.
+    #
     def parse
       return text
     end
 
+    # To avoid cutting off the email within a visible quote, wrap the
+    # `hide_everything_after` calls within a `except_in_visible_block_quotes`
+    # block:
+    #
+    #     module ExtendedEmailReplyParser
+    #       class Parsers::I18nEn < Parsers::Base
+    #         def parse
+    #           except_in_visible_block_quotes do
+    #             hide_everything_after ["From: ", "Sent: ", "To: "]
+    #           end
+    #           # ...
+    #         end
+    #       end
+    #     end
+    #
+    # Otherwise, the following email would be completely cut off after
+    # "Hi Chris,".
+    #
+    #     Hi Chris,
+    #
+    #     > From: Chris <chris@example.com>
+    #     > Sent: Saturday, July 09, 2016 3:27 PM
+    #     > To: John <john@example.com>
+    #     > Subject: The solution!
+    #     >
+    #     > Hi John,
+    #     > I've just found a solution to our big problem!
+    #
+    #     this is great, thanks!
+    #     Cheers, John
+    #
+    def except_in_visible_block_quotes(&block)
+      @email.except_in_visible_block_quotes(&block)
+      return @email.visible_text
+    end
+
+    # Boil quote like these
+    #
+    #    > Hi,
+    #
+    #    > how are you doing?
+    #
+    #    > Cheers
+    #
+    # down to
+    #
+    #    > Hi,
+    #    > how are you doing?
+    #    > Cheers
+    #
+    #
+    def remove_empty_lines_between_block_quote_lines
+      @email.remove_empty_lines_between_block_quote_lines
+      return @email.visible_text
+    end
+
+    # Some email clients do not quote the previous conversation.
+    #
+    #     Hi Chris,
+    #     this is great, thanks!
+    #     Cheers, John
+    #
+    #
+    #     From: Chris <chris@example.com>
+    #     Sent: Saturday, July 09, 2016 3:27 PM
+    #     To: John <john@example.com>
+    #     Subject: The solution!
+    #
+    #     Hi John,
+    #     I've just found a solution to our big problem!
+    #     ...
+    #
+    # To remove the previous conversation, tell the parser expressions
+    # to identify where start of the previous conversation:
+    #
+    #     module ExtendedEmailReplyParser
+    #       class Parsers::I18nEn < Parsers::Base
+    #         def parse
+    #           except_in_visible_block_quotes do
+    #             hide_everything_after ["From: ", "Sent: ", "To: "]
+    #           end
+    #           # ...
+    #         end
+    #       end
+    #     end
+    #
+    # The parser will combine the expressions to a regex:
+    #     /(#{expressions.join(".*?")}.*?\n)/m`
+    # for example:
+    #     /(From: .*?Sent: .*?To: .*?\n)/m
+    #
+    def hide_everything_after(expressions)
+      @email.hide_everything_after(expressions)
+      return @email.visible_text
+    end
+
+    # "On ... wrote:" (English)
+    # "Am ... schrieb ...:" (German)
+    # ...
+    #
+    def self.quote_header_regexes
+      @@quote_header_regexes
+    end
+
+    # The github parser (https://github.com/github/email_reply_parser) needs to
+    # know how to identify the header line of quotes, for example
+    #
+    #     "On Tue, 2011-03-01 at 18:02 +0530, Abhishek Kona wrote"
+    #
+    # Example email:
+    #
+    #     Hi,
+    #
+    #     On Tue, 2011-03-01 at 18:02 +0530, Abhishek Kona wrote:
+    #     > Hi folks
+    #     >
+    #     > What is the best way to clear a Riak bucket of all key, values after
+    #     > running a test?
+    #     > I am currently using the Java HTTP API.
+    #
+    #     You can list the keys for the bucket and call delete for each. Or if you
+    #     put the keys (and kept track of them in your test) you can delete them
+    #     one at a time (without incurring the cost of calling list first.)
+    #
+    # By default, the github parser uses the regex `/^On .* wrote:$/` for that.
+    # To make it recognize other header lines, specify their patterns using
+    # `add_quote_header_regex`.
+    #
+    # Since this is needed by the github parser, i.e. possibly before the `parse`
+    # method of your custom parser is run, make sure to add the quote header
+    # regex in the class head:
+    #
+    #     module ExtendedEmailReplyParser
+    #       class Parsers::I18nDe < Parsers::Base
+    #         add_quote_header_regex '^Am .* schrieb.*$'
+    #         # ...
+    #       end
+    #     end
+    #
+    def self.add_quote_header_regex(regex_string)
+      @@quote_header_regexes << regex_string
+    end
 
     def self.subclasses
       ObjectSpace.each_object(Class).select { |klass| klass < self }

data/lib/extended_email_reply_parser/parsers/github.rb

@@ -1,5 +1,6 @@
 module ExtendedEmailReplyParser
   class Parsers::Github < Parsers::Base
+    add_quote_header_regex '^On .* wrote:$'
 
     def parse
       EmailReplyParser.parse_reply text

data/lib/extended_email_reply_parser/parsers/i18n_de.rb

@@ -0,0 +1,14 @@
+module ExtendedEmailReplyParser
+  class Parsers::I18nDe < Parsers::Base
+    add_quote_header_regex '^Am .* schrieb.*$'
+
+    def parse
+      remove_empty_lines_between_block_quote_lines
+      except_in_visible_block_quotes do
+        hide_everything_after ["Von: ", "Gesendet: ", "An: "]
+        hide_everything_after ["Am ", "schrieb "]
+      end
+    end
+
+  end
+end

data/lib/extended_email_reply_parser/parsers/i18n_en.rb

@@ -0,0 +1,11 @@
+module ExtendedEmailReplyParser
+  class Parsers::I18nEn < Parsers::Base
+
+    def parse
+      except_in_visible_block_quotes do
+        hide_everything_after ["From: ", "Sent: ", "To: "]
+      end
+    end
+
+  end
+end