aiwilliams-mlist 0.1.4 → 0.1.5

data/CHANGELOG

@@ -1,19 +1,34 @@
+*0.1.5 [Solid Basics] (January 17, 2009)
+
+* No longer storing the list label in the message subject field, thereby supporting cleaner viewing of threads [aiwilliams]
+* Improved handling of incoming email subjects by leaving labels alone that aren't obviously the label of the list, thereby allowing for subjects like "[Ann] My New Thing" [aiwilliams]
+* Notifying subscribers when the list indicates that they are currently blocked from posting messages to a list they are subscribed to [aiwilliams]
+* Added MList::Manager module to better define the interface of list managers. This needs to be included into list manager implementations. [aiwilliams]
+* MList::List implementors may now answer the footer content to be appended to the bottom of the text/plain part of messages. [aiwilliams]
+* List footers are stripped from text/plain part of messages before being delivered. [aiwilliams]
+* Observers of MList models which are defined in client applications now work without special instruction. [aiwilliams]
+* A first pass implementation for converting html to text using Hpricot. [aiwilliams]
+* Better thread tree, supporting message navigation within a thread through a linked list kind of approach. [aiwilliams]
+* Better parent message associating using in-reply-to, then references, then subject. [aiwilliams]
+* MList.version is hash of {:major => 0, :minor => 0, :patch => 0}, with a to_s of 'MList 0.0.0'. [aiwilliams]
+* Fixed bug where original email source content was last in TMail::Mail#to_s usage. [aiwilliams]
+
 *0.1.4 [] (January 7, 2009)
 
-* Fixed bug where default email server was not allowing for settings
-* Made subject for reply place 're:' in front of the list label
-* Added really simple tree support. Really simple.
+* Fixed bug where default email server was not allowing for settings [aiwilliams]
+* Made subject for reply place 're:' in front of the list label [aiwilliams]
+* Added really simple tree support. Really simple. [aiwilliams]
 
 *0.1.3 [] (January 7, 2009)
 
-* Generating message id as UUID
-* Allowing setting of domain for message id
-* Fixed bug in storing message id
+* Generating message id as UUID [aiwilliams]
+* Allowing setting of domain for message id [aiwilliams]
+* Fixed bug in storing message id [aiwilliams]
 
 *0.1.2 [] (January 7, 2009)
 
-* Added references header when creating a reply post.
-* Including display_name in from address of EmailPost.
-* Improved extraction of text when it is nested inside parts.
+* Added references header when creating a reply post. [aiwilliams]
+* Including display_name in from address of EmailPost. [aiwilliams]
+* Improved extraction of text when it is nested inside parts. [aiwilliams]
 
 *0.1.1 [First Working Release] (January 5, 2009)

data/README

@@ -32,34 +32,49 @@ as they seem. Alas, I go boldly forward.
 
 ==== Thread Trees
 
-We need to implement this at the database level. For now, each MList::Message
-simply points to it's parent. I believe the implementation will ultimately be
-extracted from awesome_nested_set. That's a bit of work, so for now, the
-MList::Thread provides a couple of methods, roots and children, to support the
-simplest display of the tree.
+A Thread answers a tree of messages where each message in the tree knows it's
+parent, children, previous and next message, and whether it is the root or a leaf
+in the tree. The messages are actually delegates, wrappers around the real
+message instances from the thread.
+
+This approach makes a few assumptions:
+
+ # You want the tree because you will be displaying it # Moving through a tree
+message by message should 'feel' right; next is determined by walking the tree
+depth first.
+
+It may or may not prove useful someday to have this knowledge in the form of
+something like awesome_nested_set.
 
 ==== Extracting 'from' IP address from emails is not implemented
 
 http://compnetworking.about.com/od/workingwithipaddresses/qt/ipaddressemail.htm
-  
-==== Observing MList ActiveRecord subsclasses (ie, MList::Message, MList::Thread)
 
-ActiveRecord observers are reloaded at each request in development mode. They
-will be registered with the MList models each time. Since the MList models are
-required once at initialization, there will always only be one instance of the
-model class, and therefore, many instances of the observer class (all but the
-most recent invalid, since they were undefined) registered with it.
+==== Deleting messages
+
+Mail list servers don't typically deal with this, do they? When an email is sent,
+it's sent! If you delete one, how can a reply to it be accurately injected into a
+partially broken tree?
+
+Since MList is designed to integrate with applications that want to distribute
+messages, those applications may want to delete messages. I feel, at the time of
+writing this, that the feature of deleting is NOT a problem that MList should
+attempt to address in great detail. The models are ActiveRecord descendants, so
+they can be deleted with #destroy, et. al. MList will not prevent that from
+happening. This has implications, of course.
 
-There are a number of ways to solve this, the best being the one that makes
-things 'just work' such that I can delete this part of the document. For now, do
-the following in environment.rb:
+ # MList::Thread#tree will likely break if messages are gone.
 
-* remove the observer from the Rails' config.active_record.observers list
-* after the Rails::Initializer.run block, "require 'myobserver'"
-* after that require line, "Myobserver.instance"
+In my own usage, I have opted for adding a column to my mlist_messages table,
+deleted_at. The application will make decisions based on whether that column has
+a value or not. I would suggest you implement something similar, always favoring
+leaving the records in place (paranoid delete). Since MList::MailList#messages
+(and other such has_many associations) don't know about this column, they will
+always answer collections which still include those messages.
 
-This will load the observer once, thereby only registering it once with the MList
-class you are observing.
+When an MList::MailList is destroyed, all it's MList::Messages and MList::Threads
+will be deleted (:dependent => :delete_all). If no other MList::Messages are
+referencing the MList::Email records, they will also be deleted.
 
 == SYNOPSIS:
 
@@ -73,6 +88,7 @@ You love Ruby. You want MList.
 
 You'll need some gems.
 
+* hpricot
 * uuid (macaddr also)
 * tmail
 * activesupport

data/Rakefile

@@ -17,7 +17,7 @@ begin
     s.name = 'mlist'
     s.summary = 'A Ruby mailing list library designed to be integrated into other applications.'
     s.email = 'adam@thewilliams.ws'
-    s.files = FileList["[A-Z]*", "{lib}/**/*"].exclude("tmp,**/tmp")
+    s.files = FileList["[A-Z]*", "{lib,rails}/**/*"].exclude("tmp")
     s.homepage = "http://github.com/aiwilliams/mlist"
     s.description = s.summary
     s.authors = ['Adam Williams']

data/TODO

@@ -0,0 +1,36 @@
+#html_to_text
+
+ADDRESS - Address
+BLOCKQUOTE - Block quotation
+
+DIV - Generic block-level container
+DL - Definition list
+FIELDSET - Form control group
+FORM - Interactive form
+H1 - Level-one heading
+H2 - Level-two heading
+H3 - Level-three heading
+H4 - Level-four heading
+H5 - Level-five heading
+H6 - Level-six heading
+HR - Horizontal rule
+OL - Ordered list
+P - Paragraph
+PRE - Preformatted text
+
+TABLE - Table
+  output TRs as lines, csv the TDs
+
+UL - Unordered list
+
+DD - Definition description
+DT - Definition term
+LI - List item
+
+
+a (Links)     -   Link to Somewhere[1] ---- [1] http://
+b (Bold)      -   *bold*
+i (Italic)    -   _italic_
+strong        -   *strong*
+em (emphasis) -   _emphasis_
+u (underline) -   probably do nothing, maybe something like em

data/VERSION.yml

@@ -1,4 +1,4 @@
 --- 
 :minor: 1
-:patch: 4
+:patch: 5
 :major: 0

data/lib/mlist.rb

@@ -1,5 +1,8 @@
+require 'delegate'
+
 require 'uuid'
 require 'tmail'
+require 'hpricot'
 require 'activesupport'
 require 'activerecord'
 
@@ -14,7 +17,17 @@ require 'mlist/email_subscriber'
 require 'mlist/server'
 require 'mlist/thread'
 
+require 'mlist/manager'
+
 module MList
+  mattr_reader :version
+  @@version = YAML.load_file(File.join(File.dirname(__FILE__), '..', "VERSION.yml"))
+  class << @@version
+    def to_s
+      @to_s ||= [self[:major], self[:minor], self[:patch]].join('.')
+    end
+  end
+  
   class DoubleDeliveryError < StandardError
     def initialize(message)
       super("A message should never be delivered more than once. An attempt was made to deliver this message:\n#{message.inspect}")

data/lib/mlist/email.rb

@@ -28,33 +28,9 @@ module MList
       tmail.header_string('to') =~ /mlist-/
     end
     
-    # Extracts the parent message identifier from the source, using
-    # in-reply-to first, then references. Brackets around the identifier are
-    # removed.
-    #
-    # If you provide an MList::MailList, it will be searched for a message
-    # having the same subject as a last resort.
-    #
-    def parent_identifier(mail_list = nil)
-      identifier = tmail.header_string('in-reply-to') || begin
-        references = tmail['references']
-        references.ids.first if references
-      end
-      
-      if identifier
-        remove_brackets(identifier)
-      elsif mail_list && subject =~ /(^|[^\w])re:/i
-        parent_message = mail_list.messages.find(:first,
-          :select => 'identifier',
-          :conditions => ['mlist_messages.subject = ?', remove_regard(subject)],
-          :order => 'created_at asc')
-        parent_message.identifier if parent_message
-      end
-    end
-    
     def tmail=(tmail)
       @tmail = tmail
-      write_attribute(:source, tmail.to_s)
+      write_attribute(:source, tmail.port.read_all)
     end
     
     def tmail
@@ -65,11 +41,15 @@ module MList
     # methods, excluding those overridden by this Class and the [] method (an
     # ActiveRecord method).
     def method_missing(symbol, *args, &block) # :nodoc:
-      if symbol.to_s !~ /=\Z/ && symbol != :[] && symbol != :source &&  tmail.respond_to?(symbol)
+      if symbol.to_s !~ /=\Z/ && symbol != :[] && symbol != :source && tmail.respond_to?(symbol)
         tmail.__send__(symbol, *args, &block)
       else
         super
       end
     end
+    
+    def respond_to?(method)
+      super || (method.to_s !~ /=\Z/ && tmail.respond_to?(method))
+    end
   end
 end

data/lib/mlist/email_post.rb

@@ -57,6 +57,10 @@ module MList
       @attributes['subject'] || (reply_to_message ? "Re: #{reply_to_message.subject}" : nil)
     end
     
+    def to_s
+      to_tmail.to_s
+    end
+    
     def to_tmail
       raise ActiveRecord::RecordInvalid.new(self) unless valid?
       

data/lib/mlist/list.rb

@@ -5,6 +5,9 @@ module MList
   # in processing email coming to a list - that is, whatever you include this
   # into may re-define behavior appropriately.
   #
+  # Your 'subscriber' instances MUST respond to :email_address. They may
+  # optionally respond to :display_name.
+  #
   module List
     
     # Answers whether this list is active or not. All lists are active all the
@@ -14,53 +17,116 @@ module MList
       true
     end
     
-    def host
-      address.match(/@(.*)\Z/)[1]
+    # Answers whether the subscriber is blocked from posting or not. This will
+    # not be asked when the list is not active (answers _active?_ as false).
+    #
+    def blocked?(subscriber)
+      false
+    end
+    
+    # Answers the footer content for this list. Default implementation is very
+    # simple right now. Expect improvements in the future.
+    #
+    def footer_content(message)
+      %Q{The "#{label}" mailing list\nTo post messages, send email to #{post_url}}
     end
     
+    # Answer a suitable label for the list, which will be used in various
+    # parts of content that is delivered to subscribers, etc.
+    #
+    def label
+      raise 'answer the list label'
+    end
+    
+    # Answers the headers that are to be included in the emails delivered for
+    # this list. Any entries that have a nil value will not be included in the
+    # delivered email.
+    #
     def list_headers
       {
         'list-id'          => list_id,
-        'list-archive'     => (archive_url rescue nil),
-        'list-subscribe'   => (subscribe_url rescue nil),
-        'list-unsubscribe' => (unsubscribe_url rescue nil),
-        'list-owner'       => (owner_url rescue nil),
-        'list-help'        => (help_url rescue nil),
+        'list-archive'     => archive_url,
+        'list-subscribe'   => subscribe_url,
+        'list-unsubscribe' => unsubscribe_url,
+        'list-owner'       => owner_url,
+        'list-help'        => help_url,
         'list-post'        => post_url
       }
     end
     
+    # Answers a unique, never changing value for this list.
+    #
     def list_id
       raise 'answer a unique, never changing value'
     end
     
-    def name
-      address.match(/\A(.*?)@/)[1]
+    # The web address where an archive of this list may be found, nil if there
+    # is no archive.
+    #
+    def archive_url
+      nil
+    end
+    
+    # The web address of the list help site, nil if this is not supported.
+    #
+    def help_url
+      nil
+    end
+    
+    # The email address of the list owner, nil if this is not supported.
+    #
+    def owner_url
+      nil
     end
     
+    # The email address where posts should be sent. Defaults to the address of
+    # the list.
+    #
     def post_url
       address
     end
     
-    # A list is responsible for answering the recipient subscribers.
+    # The web url where subscriptions to this list may be created, nil if this
+    # is not supported.
+    #
+    def subscribe_url
+      nil
+    end
     
+    # The web url where subscriptions to this list may be deleted, nil if this
+    # is not supported.
+    #
+    def unsubscribe_url
+      nil
+    end
+    
+    # A list is responsible for answering the recipient subscribers.
+    #
     # The subscriber of the incoming message is provided if the list would
     # like to exclude it from the returned list. It is not assumed that it
     # will be included or excluded, thereby allowing the list to decide. This
     # default implementation does not include the sending subscriber in the
     # list of recipients.
     #
-    # Your 'subscriber' instance MUST respond to :email_address. They may
-    # optionally respond to :display_name.
-    #
     def recipients(subscriber)
       subscribers.reject {|s| s.email_address == subscriber.email_address}
     end
     
+    # A list must answer the subscriber who's email address is that of the one
+    # provided. The default implementation will pick the first instance that
+    # answers subscriber.email_address == email_address. Your implementation
+    # should probably select just one record.
+    #
     def subscriber(email_address)
       subscribers.detect {|s| s.email_address == email_address}
     end
     
+    # A list must answer whether there is a subscriber who's email address is
+    # that of the one provided. This is checked before the subscriber is
+    # requested in order to allow for the lightest weight check possible; that
+    # is, your implementation could avoid loading the actual subscriber
+    # instance.
+    #
     def subscriber?(email_address)
       !subscriber(email_address).nil?
     end
@@ -69,6 +135,15 @@ module MList
     # certain events occur during the processing of email sent to a list.
     #
     module Callbacks
+      
+      # Called when an email is a post to the list by a subscriber whom the
+      # list claims is blocked (answers true to _blocked?(subscriber)_). This
+      # will not be called if the list is inactive (answers false to
+      # _active?_);
+      #
+      def blocked_subscriber_post(email, subscriber)
+      end
+      
       def bounce(email)
       end
       

data/lib/mlist/mail_list.rb

@@ -18,8 +18,11 @@ module MList
       mail_list
     end
     
+    include MList::Util::EmailHelpers
+    
     belongs_to :manager_list, :polymorphic => true
     
+    before_destroy :delete_unreferenced_email
     has_many :messages, :class_name => 'MList::Message', :dependent => :delete_all
     has_many :threads, :class_name => 'MList::Thread', :dependent => :delete_all
     
@@ -39,14 +42,13 @@ module MList
         :mail_list => self,
         :subscriber => email.subscriber,
         :recipients => list.recipients(email.subscriber),
-        :email => MList::Email.new(:tmail => email.to_tmail)
+        :email => MList::Email.new(:source => email.to_s)
       ), :search_parent => false
     end
     
     # Processes the email received by the MList::Server.
     #
-    def process_email(email)
-      subscriber = list.subscriber(email.from_address)
+    def process_email(email, subscriber)
       recipients = list.recipients(subscriber)
       process_message messages.build(
         :mail_list => self,
@@ -56,6 +58,46 @@ module MList
       )
     end
     
+    # Answers the provided subject with superfluous 're:' and this list's
+    # labels removed.
+    #
+    #   clean_subject('[List Label] Re: The new Chrome Browser from Google') => 'Re: The new Chrome Browser from Google'
+    #   clean_subject('Re: [List Label] Re: The new Chrome Browser from Google') => 'Re: The new Chrome Browser from Google'
+    #
+    def clean_subject(string)
+      without_label = string.gsub(subject_prefix_regex, '')
+      if without_label =~ REGARD_RE
+        "Re: #{remove_regard(without_label)}"
+      else
+        without_label
+      end
+    end
+    
+    def find_parent_message(email)
+      if in_reply_to = email.header_string('in-reply-to')
+        message = messages.find(:first,
+          :conditions => ['identifier = ?', remove_brackets(in_reply_to)])
+        return message if message
+      end
+      
+      if email.references
+        reference_identifiers = email.references.collect {|rid| remove_brackets(rid)}
+        message = messages.find(:first,
+          :conditions => ['identifier in (?)', reference_identifiers],
+          :order => 'created_at desc')
+        return message if message
+      end
+      
+      if email.subject =~ REGARD_RE
+        message = messages.find(:first,
+          :conditions => ['subject = ?', remove_regard(clean_subject(email.subject))],
+          :order => 'created_at asc')
+        return message if message
+      end
+    end
+    
+    # The MList::List instance of the list manager.
+    #
     def list
       @list ||= manager_list
     end
@@ -75,17 +117,59 @@ module MList
       !message.recipients.blank?
     end
     
+    # Distinct footer start marker. It is important to realize that changing
+    # this could be problematic.
+    #
+    FOOTER_BLOCK_START    = "-~----~~----~----~----~----~---~~-~----~------~--~-~-"
+    
+    # Distinct footer end marker. It is important to realize that changing
+    # this could be problematic.
+    #
+    FOOTER_BLOCK_END      = "--~--~---~-----~--~----~-----~~~----~---~---~--~----~"
+    
     private
+      FOOTER_BLOCK_START_RE = %r[#{FOOTER_BLOCK_START}]
+      FOOTER_BLOCK_END_RE   = %r[#{FOOTER_BLOCK_END}]
+      
       # http://mail.python.org/pipermail/mailman-developers/2006-April/018718.html
       def bounce_headers
-        {'sender'    => "mlist-#{address}",
-         'errors-to' => "mlist-#{address}"}
+        # tmail would not correctly quote the label in the sender header, which would break smtp delivery
+        {'sender' => "<mlist-#{address}>", 'errors-to' => "#{label} <mlist-#{address}>"}
+      end
+      
+      def delete_unreferenced_email
+        conditions = %Q{
+          mlist_emails.id in (
+            select me.id from mlist_emails me left join mlist_messages mm on mm.email_id = me.id
+            where mm.mail_list_id = #{id}
+          ) AND mlist_emails.id not in (
+            select meb.id from mlist_emails meb left join mlist_messages mmb on mmb.email_id = meb.id
+            where mmb.mail_list_id != #{id}
+          )}
+        MList::Email.delete_all(conditions)
+      end
+      
+      def strip_list_footers(content)
+        if content =~ FOOTER_BLOCK_START_RE
+          in_footer_block = false
+          content = normalize_new_lines(content)
+          content = content.split("\n").reject do |line|
+            if in_footer_block
+              in_footer_block = line !~ FOOTER_BLOCK_END_RE
+              true
+            else
+              in_footer_block = line =~ FOOTER_BLOCK_START_RE
+            end
+          end.join("\n").rstrip
+        end
+        content
       end
       
       # http://www.jamesshuggins.com/h/web1/list-email-headers.htm
       def list_headers
-        headers = list.list_headers
+        headers = list.list_headers.dup
         headers['x-beenthere'] = address
+        headers['x-mlist-version'] = MList.version.to_s
         headers.update(bounce_headers)
         headers.delete_if {|k,v| v.nil?}
       end
@@ -118,36 +202,55 @@ module MList
       def prepare_delivery(message, options)
         message.identifier = outgoing_server.generate_message_id
         message.created_at = options[:delivery_time]
+        message.subject = clean_subject(message.subject)
         returning(message.delivery) do |delivery|
           delivery.date = message.created_at
           delivery.message_id = message.identifier
           delivery.mailer = message.mailer
           delivery.headers = list_headers
-          delivery.subject = list_subject(message)
+          delivery.subject = list_subject(message.subject)
           delivery.to = address
           delivery.bcc = message.recipients.collect(&:email_address)
           delivery.reply_to = "#{label} <#{post_url}>"
+          prepare_list_footer(delivery, message)
         end
       end
       
-      def list_subject(message)
-        prefix = "[#{label}]"
-        subject = message.subject.dup
-        if subject =~ /re: /i
-          subject.gsub!(%r{(re:\s*)}i, '')
-          prefix = 'Re: ' + prefix
+      def prepare_list_footer(delivery, message)
+        text_plain_part = delivery.text_plain_part
+        return unless text_plain_part
+        
+        content = strip_list_footers(text_plain_part.body)
+        content << "\n\n" unless content.end_with?("\n\n")
+        content << list_footer(message)
+        text_plain_part.body = content
+      end
+      
+      def list_footer(message)
+        content = list.footer_content(message)
+        "#{FOOTER_BLOCK_START}\n#{content}\n#{FOOTER_BLOCK_END}"
+      end
+      
+      def list_subject(string)
+        list_subject = string.dup
+        if list_subject =~ REGARD_RE
+          "Re: #{subject_prefix} #{remove_regard(list_subject)}"
+        else
+          "#{subject_prefix} #{list_subject}"
         end
-        subject.gsub!(/\[.*?\]/, '')
-        subject.strip!
-        "#{prefix} #{subject}"
       end
       
       def find_thread(message, options)
-        if options[:search_parent]
-          message.parent_identifier = message.email.parent_identifier(self)
-          message.parent = messages.find_by_identifier(message.parent_identifier)
-        end
+        message.parent = find_parent_message(message.email) if message.email && options[:search_parent]
         message.parent ? message.parent.thread : threads.build
       end
+      
+      def subject_prefix_regex
+        @subject_prefix_regex ||= Regexp.new(Regexp.escape(subject_prefix) + ' ')
+      end
+      
+      def subject_prefix
+        @subject_prefix ||= "[#{label}]"
+      end
   end
 end

data/lib/mlist/manager.rb

@@ -0,0 +1,30 @@
+require 'mlist/manager/notifier'
+
+module MList
+  
+  # The interface of list managers.
+  #
+  # A module is provided instead of a base class to allow implementors to
+  # subclass whatever they like. Practically speaking, they can create an
+  # ActiveRecord subclass.
+  #
+  module Manager
+    
+    # Answers an enumeration of MList::List implementations to which the given
+    # email should be published.
+    #
+    def lists(email)
+      raise 'implement in your list manager'
+    end
+    
+    # Answers the MList::Manager::Notifier of this list manager. Includers of
+    # this module may initialize the @notifier instance variable with their
+    # own implementation/subclass to generate custom content for the different
+    # notices.
+    #
+    def notifier
+      @notifier ||= MList::Manager::Notifier.new
+    end
+  end
+  
+end

data/lib/mlist/manager/database.rb

@@ -2,6 +2,8 @@ module MList
   module Manager
     
     class Database
+      include ::MList::Manager
+      
       def create_list(address, attributes = {})
         attributes = {
           :address => address,
@@ -16,7 +18,7 @@ module MList
       end
       
       def no_lists_found(email)
-        # your application may care
+        # TODO: Move to notifier
       end
       
       class List < ActiveRecord::Base
@@ -24,6 +26,10 @@ module MList
         
         has_many :subscribers, :dependent => :delete_all
         
+        def label
+          self[:label]
+        end
+        
         def list_id
           "#{self.class.name}#{id}"
         end

data/lib/mlist/manager/notifier.rb

@@ -0,0 +1,31 @@
+module MList
+  module Manager
+    
+    # Constructs the notices that are sent to list subscribers. Applications
+    # may subclass this to customize the content of a notice delivery.
+    #
+    class Notifier
+      
+      # Answers the delivery that will be sent to a subscriber when an
+      # MList::List indicates that the distribution of an email from that
+      # subscriber has been blocked.
+      #
+      def subscriber_blocked(list, email, subscriber)
+        delivery = MList::Util::TMailBuilder.new(TMail::Mail.new)
+        delivery.write_header('x-mlist-loop', 'notice')
+        delivery.write_header('x-mlist-notice', 'subscriber_blocked')
+        delivery.to = subscriber.email_address
+        delivery.from = "mlist-#{list.address}"
+        prepare_subscriber_blocked_content(list, email, subscriber, delivery)
+        delivery
+      end
+      
+      protected
+        def prepare_subscriber_blocked_content(list, email, subscriber, delivery)
+          delivery.set_content_type('text/plain')
+          delivery.body = %{Although you are a subscriber to this list, your message cannot be posted at this time. Please contact the administrator of the list.}
+        end
+    end
+    
+  end
+end

data/lib/mlist/message.rb

@@ -10,6 +10,8 @@ module MList
     belongs_to :mail_list, :class_name => 'MList::MailList', :counter_cache => :messages_count
     belongs_to :thread, :class_name => 'MList::Thread', :counter_cache => :messages_count
     
+    after_destroy :delete_unreferenced_email
+    
     # A temporary storage of recipient subscribers, obtained from
     # MList::Lists. This list is not available when a message is reloaded.
     #
@@ -73,13 +75,29 @@ module MList
       text_to_html(text_for_reply)
     end
     
-    # Answers the subject with all prefixes removed.
+    def parent_with_identifier_capture=(parent)
+      if parent
+        self.parent_without_identifier_capture = parent
+        self.parent_identifier = parent.identifier
+      else
+        self.parent_without_identifier_capture = nil
+        self.parent_identifier = nil
+      end
+    end
+    alias_method_chain :parent=, :identifier_capture
+    
+    # Answers the subject with 'Re:' prefixed. Note that it is the
+    # responsibility of the MList::MailList to perform any processing of the
+    # persisted subject (ie, cleaning up labels, etc).
+    #
+    #   message.subject = '[List Label] Re: The new Chrome Browser from Google'
+    #   message.subject_for_reply   => 'Re: [List Label] The new Chrome Browser from Google'
     #
     #   message.subject = 'Re: [List Label] Re: The new Chrome Browser from Google'
-    #   message.subject_for_reply   => 'Re: The new Chrome Browser from Google'
+    #   message.subject_for_reply   => 'Re: [List Label] The new Chrome Browser from Google'
     #
     def subject_for_reply
-      "Re: #{remove_regard(subject)}"
+      subject =~ REGARD_RE ? subject : "Re: #{subject}"
     end
     
     # Answers the subscriber from which this message comes.
@@ -113,5 +131,10 @@ module MList
         @subscriber = self.subscriber_address = self.subscriber_type = self.subscriber_id = nil
       end
     end
+    
+    private
+      def delete_unreferenced_email
+        email.destroy unless MList::Message.count(:conditions => "email_id = #{email_id} and id != #{id}") > 0
+      end
   end
 end

data/lib/mlist/server.rb

@@ -1,10 +1,11 @@
 module MList
   class Server
-    attr_reader :list_manager, :email_server
+    attr_reader :list_manager, :email_server, :notifier
     
     def initialize(config)
       @list_manager = config[:list_manager]
       @email_server = config[:email_server]
+      @notifier = MList::Manager::Notifier.new
       @email_server.receiver(self)
     end
     
@@ -24,23 +25,38 @@ module MList
     end
     
     protected
-      def process_bounce(list, email)
-        list.bounce(email)
-      end
-      
       def process_post(lists, email)
         lists.each do |list|
           next if email.been_here?(list)
           if list.subscriber?(email.from_address)
-            if list.active?
-              mail_list(list).process_email(email)
-            else
-              list.inactive_post(email)
-            end
+            publish_if_list_active(list, email)
           else
             list.non_subscriber_post(email)
           end
         end
       end
+      
+      def publish_if_list_active(list, email)
+        if list.active?
+          subscriber = list.subscriber(email.from_address)
+          publish_unless_blocked(list, email, subscriber)
+        else
+          list.inactive_post(email)
+        end
+      end
+      
+      def publish_unless_blocked(list, email, subscriber)
+        if list.blocked?(subscriber)
+          notice_delivery = notifier.subscriber_blocked(list, email, subscriber)
+          email_server.deliver(notice_delivery.tmail)
+        else
+          mail_list(list).process_email(email, subscriber)
+        end
+      end
+      
+      def process_bounce(list, email)
+        list.bounce(email)
+      end
+      
   end
 end

data/lib/mlist/thread.rb

@@ -5,34 +5,94 @@ module MList
     belongs_to :mail_list, :class_name => 'MList::MailList', :counter_cache => :threads_count
     has_many :messages, :class_name => 'MList::Message', :dependent => :delete_all
     
-    def children(message)
-      messages.select {|m| m.parent == message}
-    end
-    
     def first?(message)
-      messages.first == message
+      tree_order.first == message
     end
     
     def last?(message)
-      messages.last == message
+      tree_order.last == message
     end
     
     def next(message)
-      i = messages.index(message)
+      i = tree_order.index(message)
       messages[i + 1] unless messages.size < i
     end
     
     def previous(message)
-      i = messages.index(message)
+      i = tree_order.index(message)
       messages[i - 1] if i > 0
     end
     
-    def roots
-      messages.select {|m| m.parent.nil?}
-    end
-    
     def subject
       messages.first.subject
     end
+    
+    # Answers a tree of messages.
+    #
+    # The nodes of the tree are decorated to act like a linked list, providing
+    # pointers to _next_ and _previous_ in the tree.
+    #
+    def tree
+      return nil if messages.size == 0
+      build_tree unless @tree
+      @tree
+    end
+    
+    def tree_order
+      build_tree unless @tree
+      @tree_order
+    end
+    
+    private
+      def build_tree
+        nodes = messages.collect do |m|
+          m.parent = messages.detect {|pm| pm.id == m.parent_id}
+          Node.new(m)
+        end
+        
+        nodes.each do |node|
+          if parent_node = nodes.detect {|n| n == node.parent}
+            node.parent_node = parent_node
+            parent_node.children << node
+          end
+        end
+        
+        @tree_order = []
+        previous_node = nil
+        nodes.first.visit do |node|
+          @tree_order << node
+          if previous_node
+            node.previous = previous_node
+            previous_node.next = node
+          end
+          previous_node = node
+        end
+        
+        @tree = @tree_order.first
+      end
+      
+      class Node < DelegateClass(Message)
+        attr_accessor :parent_node, :previous, :next
+        attr_reader :children
+        
+        def initialize(message)
+          super
+          @children = []
+        end
+        
+        def leaf?
+          children.empty?
+        end
+        
+        def root?
+          parent_node.nil?
+        end
+        
+        def visit(&visitor)
+          visitor.call self
+          children.each {|c| c.visit(&visitor)}
+        end
+      end
+      
   end
 end

data/lib/mlist/util/email_helpers.rb

@@ -1,6 +1,83 @@
 module MList
   module Util
     
+    class HtmlTextExtraction
+      def initialize(html)
+        @doc = Hpricot(html)
+      end
+      
+      def execute
+        @text, @anchors = '', []
+        @doc.each_child do |node|
+          extract_text_from_node(node) if Hpricot::Elem::Trav === node
+        end
+        @text.strip!
+        unless @anchors.empty?
+          refs = []
+          @anchors.each_with_index do |href, i|
+            refs << "[#{i+1}] #{href}"
+          end
+          @text << "\n\n--\n#{refs.join("\n")}"
+        end
+        @text
+      end
+      
+      def extract_text_from_node(node)
+        case node.name
+        when 'head'
+        when 'h1', 'h2', 'h3', 'h4', 'h5', 'h6'
+          @text << node.inner_text
+          @text << "\n\n"
+        when 'br'
+          @text << "\n"
+        when 'ol'
+          node.children_of_type('li').each_with_index do |li, i|
+            @text << " #{i+1}. #{li.inner_text}"
+            @text << "\n\n"
+          end
+        when 'ul'
+          node.children_of_type('li').each do |li|
+            @text << " * #{li.inner_text.strip}"
+            @text << "\n\n"
+          end
+        when 'strong'
+          @text << "*#{node.inner_text}*"
+        when 'em'
+          @text << "_#{node.inner_text}_"
+        when 'dl'
+          node.traverse_element('dt', 'dd') do |dt_dd|
+            extract_text_from_node(dt_dd)
+          end
+        when 'a'
+          @anchors << node['href']
+          extract_text_from_text_node(node)
+          @text << "[#{@anchors.size}]"
+        when 'p', 'dt', 'dd'
+          extract_text_from_children(node)
+          @text.rstrip!
+          @text << "\n\n"
+        else
+          extract_text_from_children(node)
+        end
+      end
+      
+      def extract_text_from_children(elem)
+        elem.each_child do |node|
+          case node
+          when Hpricot::Text::Trav
+            extract_text_from_text_node(node)
+          when Hpricot::Elem::Trav
+            extract_text_from_node(node)
+          end
+        end
+      end
+      
+      def extract_text_from_text_node(node)
+        text = @text.end_with?("\n") ? node.inner_text.lstrip : node.inner_text
+        @text << text.gsub(/\s{2,}/, ' ').sub(/\n/, '')
+      end
+    end
+    
     module EmailHelpers
       def sanitize_header(charset, name, *values)
         header_sanitizer(name).call(charset, *values)
@@ -10,12 +87,15 @@ module MList
         Util.default_header_sanitizers[name]
       end
       
+      def html_to_text(html)
+        HtmlTextExtraction.new(html).execute
+      end
+      
       def normalize_new_lines(text)
         text.to_s.gsub(/\r\n?/, "\n")
       end
       
       BRACKETS_RE = /\A<(.*?)>\Z/
-      
       def bracket(string)
         string.blank? || string =~ BRACKETS_RE ? string : "<#{string}>"
       end
@@ -24,9 +104,12 @@ module MList
         string =~ BRACKETS_RE ? $1 : string
       end
       
+      REGARD_RE = /(^|[^\w])re: /i
       def remove_regard(string)
-        stripped = string.strip
-        stripped =~ /\A.*re:\s+(\[.*\]\s*)?(.*?)\Z/i ? $2 : stripped
+        while string =~ REGARD_RE
+          string = string.sub(REGARD_RE, ' ')
+        end
+        string.strip
       end
       
       def text_to_html(text)

data/lib/mlist/util/tmail_methods.rb

@@ -19,8 +19,8 @@ module MList
         when 'text/html'
           tmail.body.strip
         when 'multipart/alternative'
-          text_part = tmail.parts.detect {|part| part.content_type == 'text/html'}
-          text_part.body.strip if text_part
+          html_part = tmail.parts.detect {|part| part.content_type == 'text/html'}
+          html_part.body.strip if html_part
         end
       end
       
@@ -33,19 +33,45 @@ module MList
       end
       
       def text
-        returning('') {|content| extract_text_content(tmail, content)}
+        text_content = ''
+        extract_text_content(tmail, text_content)
+        return text_content unless text_content.blank?
+        
+        html_content = ''
+        extract_html_content(tmail, html_content)
+        return html_to_text(html_content) unless html_content.blank?
+        
+        return nil
+      end
+      
+      # Answers the first text/plain part it can find, the tmail itself if
+      # it's content type is text/plain.
+      #
+      def text_plain_part(part = tmail)
+        case part.content_type
+        when 'text/plain'
+          part
+        when 'multipart/alternative'
+          part.parts.detect {|part| text_plain_part(part)}
+        end
       end
       
       private
+        def extract_html_content(part, collector)
+          case part.content_type
+          when 'text/html'
+            collector << part.body.strip
+          when 'multipart/alternative', 'multipart/mixed', 'multipart/related'
+            part.parts.each {|part| extract_html_content(part, collector)}
+          end
+        end
+        
         def extract_text_content(part, collector)
           case part.content_type
           when 'text/plain'
             collector << part.body.strip
-          when 'multipart/alternative'
-            text_part = part.parts.detect {|part| part.content_type == 'text/plain'}
-            collector << text_part.body.strip if text_part
-          when 'multipart/mixed', 'multipart/related'
-            part.parts.each {|mixed_part| extract_text_content(mixed_part, collector)}
+          when 'multipart/alternative', 'multipart/mixed', 'multipart/related'
+            part.parts.each {|part| extract_text_content(part, collector)}
           end
         end
     end

data/rails/init.rb

@@ -0,0 +1,28 @@
+require 'dispatcher' unless defined?(::Dispatcher)
+
+Dispatcher.module_eval do
+  # Provides the mechinism to support applications that want to observe MList
+  # models.
+  #
+  # ActiveRecord observers are reloaded at each request in development mode.
+  # They will be registered with the MList models each time. Since the MList
+  # models are required once at initialization, there will always only be one
+  # instance of the model class, and therefore, many instances of the observer
+  # class registered with it; all but the most recent are invalid, since they
+  # were undefined when the dispatcher reloaded the application.
+  #
+  # Why not an initializer "to_prepare" block? Simply because we must clear
+  # the observers in the ActiveRecord classes before the
+  # ActiveRecord::Base.instantiate_observers call is made by the prepare block
+  # that we cannot get in front of with the initializer approach. Also, it
+  # lessens the configuration burden of the MList client application.
+  #
+  # Should we ever have observers in MList, this will likely need more careful
+  # attention.
+  #    
+  def reload_application_with_plugin_record_support
+    ActiveRecord::Base.send(:subclasses).each(&:delete_observers)
+    reload_application_without_plugin_record_support
+  end
+  alias_method_chain :reload_application, :plugin_record_support
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: aiwilliams-mlist
 version: !ruby/object:Gem::Version 
-  version: 0.1.4
+  version: 0.1.5
 platform: ruby
 authors: 
 - Adam Williams
@@ -9,7 +9,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2009-02-09 00:00:00 -08:00
+date: 2009-02-17 00:00:00 -08:00
 default_executable: 
 dependencies: []
 
@@ -25,6 +25,7 @@ files:
 - CHANGELOG
 - Rakefile
 - README
+- TODO
 - VERSION.yml
 - lib/mlist
 - lib/mlist/email.rb
@@ -41,6 +42,8 @@ files:
 - lib/mlist/mail_list.rb
 - lib/mlist/manager
 - lib/mlist/manager/database.rb
+- lib/mlist/manager/notifier.rb
+- lib/mlist/manager.rb
 - lib/mlist/message.rb
 - lib/mlist/server.rb
 - lib/mlist/thread.rb
@@ -53,6 +56,7 @@ files:
 - lib/mlist/util.rb
 - lib/mlist.rb
 - lib/pop_ssl.rb
+- rails/init.rb
 has_rdoc: false
 homepage: http://github.com/aiwilliams/mlist
 post_install_message: