mlist 0.1.9

data/CHANGELOG

@@ -0,0 +1,59 @@
+*0.1.9 [Enhancement] (2009-12-21)
+
+* DomainKey-Signature and DKIM-Signature headers will not be published to list so that sending SMTP servers may sign. [aiwilliams]
+* Deploying to http://gemcutter.org [aiwilliams]
+
+*0.1.8 [Enhancement] (2009-08-19)
+
+* return-receipt-to header will not be published to list to avoid having recipients spam list with receipts [aiwilliams]
+
+*0.1.7 [Bug Fixes, Delivery Improvements] (2009-08-14)
+
+*   is maintained as spacing in html_to_text conversions [aiwilliams]
+* Fixed bug where delivered email included the original Cc header. This could cause all kinds of problems. [aiwilliams]
+* Fixed bug where list addresses in Cc header were not be utilized in determining the lists to deliver to. [aiwilliams]
+* Fixed bug where delivery date was left to the mercy of the Rails time configuration. Using Time.now. [aiwilliams]
+* Manager list can indicate if reply-to should be list address or subscriber. [aiwilliams]
+* Leaving the reply-to field intact when it already exists on incoming email. [aiwilliams]
+* Messages will not be delivered to addresses found in the TO and CC fields in order to keep those recipients from getting two emails. [aiwilliams]
+* Added ability to optionally copy sender. [aiwilliams]
+
+*0.1.6 [Bug Fixes] (March 5, 2009)
+
+* Messages are processed even when there are no recipients [aiwilliams]
+* Escaping DQUOTE and \ in email address phrase for some headers [aiwilliams]
+
+*0.1.5 [Solid Basics] (January 17, 2009)
+
+* No longer storing the list label in the message subject field, thereby supporting cleaner viewing of threads [aiwilliams]
+* Improved handling of incoming email subjects by leaving labels alone that aren't obviously the label of the list, thereby allowing for subjects like "[Ann] My New Thing" [aiwilliams]
+* Notifying subscribers when the list indicates that they are currently blocked from posting messages to a list they are subscribed to [aiwilliams]
+* Added MList::Manager module to better define the interface of list managers. This needs to be included into list manager implementations. [aiwilliams]
+* MList::List implementors may now answer the footer content to be appended to the bottom of the text/plain part of messages. [aiwilliams]
+* List footers are stripped from text/plain part of messages before being delivered. [aiwilliams]
+* Observers of MList models which are defined in client applications now work without special instruction. [aiwilliams]
+* A first pass implementation for converting html to text using Hpricot. [aiwilliams]
+* Better thread tree, supporting message navigation within a thread through a linked list kind of approach. [aiwilliams]
+* Better parent message associating using in-reply-to, then references, then subject. [aiwilliams]
+* MList.version is hash of {:major => 0, :minor => 0, :patch => 0}, with a to_s of 'MList 0.0.0'. [aiwilliams]
+* Fixed bug where original email source content was last in TMail::Mail#to_s usage. [aiwilliams]
+
+*0.1.4 [] (January 7, 2009)
+
+* Fixed bug where default email server was not allowing for settings [aiwilliams]
+* Made subject for reply place 're:' in front of the list label [aiwilliams]
+* Added really simple tree support. Really simple. [aiwilliams]
+
+*0.1.3 [] (January 7, 2009)
+
+* Generating message id as UUID [aiwilliams]
+* Allowing setting of domain for message id [aiwilliams]
+* Fixed bug in storing message id [aiwilliams]
+
+*0.1.2 [] (January 7, 2009)
+
+* Added references header when creating a reply post. [aiwilliams]
+* Including display_name in from address of EmailPost. [aiwilliams]
+* Improved extraction of text when it is nested inside parts. [aiwilliams]
+
+*0.1.1 [First Working Release] (January 5, 2009)

data/README

@@ -0,0 +1,204 @@
+= MList
+
+== DESCRIPTION:
+
+An insane attempt to build a mail list server library that can be used in other
+applications very easily. The first target is Ruby applications that can load
+MList models for direct, embedded integration. It will later have a RESTful API
+so that any language/architecture can be easily integrated. That may depend
+heavily on community involvement...
+
+== FEATURES/PROBLEMS:
+
+If you have any experience with mailing lists, things can be understood most
+quickly with this: MList is the mailing list server and database, your
+application is the list manager.
+
+MList is being used in a production environment as a Ruby gem/Rails plugin. I
+decided not to use other list software primarily because a) we already had a
+running, sophisticated list manager that I didn't care to synchronize with
+something else, b) we have an exceptional UI design for listing messages in
+threads and ultimately will have search and c) integration options for the other
+software looked to be a problem in themselves.
+
+And then there's the fame and glory that comes with building something like
+this...
+
+There is a LOT to do: segmenting, i18n, backscatter - only the Mailman developers
+know what else. I have enough experience to know that rewrites are NEVER as easy
+as they seem. Alas, I go boldly forward.
+
+==== Thread Trees
+
+A Thread answers a tree of messages where each message in the tree knows it's
+parent, children, previous and next message, and whether it is the root or a leaf
+in the tree. The messages are actually delegates, wrappers around the real
+message instances from the thread.
+
+This approach makes a few assumptions:
+
+ # You want the tree because you will be displaying it # Moving through a tree
+message by message should 'feel' right; next is determined by walking the tree
+depth first.
+
+It may or may not prove useful someday to have this knowledge in the form of
+something like awesome_nested_set.
+
+==== Extracting 'from' IP address from emails is not implemented
+
+http://compnetworking.about.com/od/workingwithipaddresses/qt/ipaddressemail.htm
+
+==== Deleting messages
+
+Mail list servers don't typically deal with this, do they? When an email is sent,
+it's sent! If you delete one, how can a reply to it be accurately injected into a
+partially broken tree?
+
+Since MList is designed to integrate with applications that want to distribute
+messages, those applications may want to delete messages. I feel, at the time of
+writing this, that the feature of deleting is NOT a problem that MList should
+attempt to address in great detail. The models are ActiveRecord descendants, so
+they can be deleted with #destroy, et. al. MList will not prevent that from
+happening. This has implications, of course.
+
+ # MList::Thread#tree will likely break if messages are gone.
+
+In my own usage, I have opted for adding a column to my mlist_messages table,
+deleted_at. The application will make decisions based on whether that column has
+a value or not. I would suggest you implement something similar, always favoring
+leaving the records in place (paranoid delete). Since MList::MailList#messages
+(and other such has_many associations) don't know about this column, they will
+always answer collections which still include those messages.
+
+When an MList::MailList is destroyed, all it's MList::Messages and MList::Threads
+will be deleted (:dependent => :delete_all). If no other MList::Messages are
+referencing the MList::Email records, they will also be deleted.
+
+==== Ensuring Delivery
+
+The internet email system is a mess. Spam has proven to be a significant
+problem which plagues both your inbox and the service providers who work to
+deliver email into it. So, what must you, the user of MList do, to ensure that
+your software plays nice, and your emails are delivered?
+
+Of course, this problem goes well beyond MList - any email you send from your
+application servers will suffer the consequences of a misconfigured domain.
+Here's what I've learned:
+
+ * Go to http://old.openspf.org/wizard.html and create your SPF record
+   content, to be placed in a TXT record for your domain.
+   - If you are using Google Apps, check this out:
+     http://www.google.com/support/a/bin/answer.py?hl=en&answer=33786
+ * Make sure you have an abuse@yourdomain.com and postmaster@yourdomain.com
+   address.
+   - If you are using Google Apps, you will need to create 'groups' for those
+     addresses, adding yourself as a recipient. You can read more about it
+     here: http://blog.wordtothewise.com/2009/01/google-apps-wheres-my-abuse/
+ * Be prepared to visit the Feedback Loop (FBL) configuration pages of many
+   ISPs, like the one at AOL: http://postmaster.aol.com/fbl/
+ * You'll probably want to have a way to resolve an email address (subscriber)
+   to an IP address from which the subscriber subscribed. This will help your
+   case should you have an ISP rejecting your mail.
+
+There is a great article here:
+
+  http://community-support.engineyard.com/faqs/guides/making-sure-your-email-gets-delivered
+
+Although that is EY focused, it gives you a good idea of some of the things
+you should get right.
+
+== SYNOPSIS:
+
+Let's say you want your web application to have a mailing list feature. Let's
+also say you care about the UI, and you don't want to learn all about creating
+the correct HTML structures for a mailing list. You want to have lots of power
+for searching the mail, and you have your own strategy for managing the lists.
+You love Ruby. You want MList.
+
+== REQUIREMENTS:
+
+You'll need some gems.
+
+* hpricot
+* uuid (macaddr also)
+* tmail
+* active_support
+* active_record
+
+== INSTALL:
+
+- gem sources add http://gemcutter.org
+- sudo gem install mlist
+
+For now, copy the mlist_ tables from the spec/fixtures/schema.rb file and move
+them to a new migration in your application. Run the migration.
+
+Now you'll need to create the MList::Server and provide it with a list manager
+and MList::EmailServer::Default instance. Something like this in your
+environment.rb after the initialize block (our gem needs to have been loaded):
+
+  Rails::Initializer.run do |config|
+    # Please do specify a version, and check for the latest!
+    config.gem "mlist", :version => '0.1.0'
+  end
+  
+  MLIST_SERVER = MList::Server.new(
+    :list_manager => MList::Manager::Database.new,
+    :email_server => MList::EmailServer::Default.new(
+      MList::EmailServer::Pop.new(
+        :ssl      => false,
+        :server   => 'pop.gmail.com',
+        :port     => '995',
+        :username => 'yourusername',
+        :password => 'yourpassword'
+      ),
+      MList::EmailServer::Smtp.new(
+        ActionMailer::Base.smtp_settings # probably good enough!
+      )
+    )
+  )
+
+Your list manager needs to implement only two methods. Check out (and use if you
+like) the MList::Manager::Database for more information.
+
+You'll need something to trigger the incoming server process. Take your pick from
+http://wiki.rubyonrails.org/rails/pages/HowToRunBackgroundJobsInRails. In the
+end, if you don't write your own incoming server thing and go with the POP GMail
+example above, your background process will "MLIST_SERVER.email_server.execute".
+
+Take a look at MList::EmailPost if you're building a UI. It can be posted to a
+list something like this:
+
+  class MyList # instances of these are given by your list manager
+    def post(attributes)
+      email = MList::EmailPost.new({
+        :mailer => 'MyApplication'
+      }.merge(attributes))
+      raise 'You can validate these' unless email.valid?
+      message = MLIST_SERVER.mail_list(self).post(email)
+      # do what you will with message. it's already saved.
+    end
+  end
+
+== LICENSE:
+
+(The MIT License)
+
+Copyright (c) 2008-2009 Adam Williams (aiwilliams)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the 'Software'), to deal in the
+Software without restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
+Software, and to permit persons to whom the Software is furnished to do so,
+subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
+AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/Rakefile

@@ -0,0 +1,27 @@
+$:.unshift(File.join(File.dirname(__FILE__), 'lib'))
+
+require 'rubygems'
+require 'spec/rake/spectask'
+
+task :default => :spec
+
+desc "Run all specs"
+Spec::Rake::SpecTask.new do |t|
+  t.spec_files = FileList['spec/**/*_spec.rb']
+  t.spec_opts = ['--options', 'spec/spec.opts']
+end
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |s|
+    s.name = 'mlist'
+    s.summary = 'A Ruby mailing list library designed to be integrated into other applications.'
+    s.email = 'adam@thewilliams.ws'
+    s.files = FileList["[A-Z]*", "{lib,rails}/**/*"].exclude("tmp")
+    s.homepage = "http://github.com/aiwilliams/mlist"
+    s.description = s.summary
+    s.authors = ['Adam Williams']
+  end
+rescue LoadError
+  puts "Jeweler not available. Install it with: sudo gem install technicalpickles-jeweler -s http://gems.github.com"
+end

data/TODO

@@ -0,0 +1,36 @@
+#html_to_text
+
+ADDRESS - Address
+BLOCKQUOTE - Block quotation
+
+DIV - Generic block-level container
+DL - Definition list
+FIELDSET - Form control group
+FORM - Interactive form
+H1 - Level-one heading
+H2 - Level-two heading
+H3 - Level-three heading
+H4 - Level-four heading
+H5 - Level-five heading
+H6 - Level-six heading
+HR - Horizontal rule
+OL - Ordered list
+P - Paragraph
+PRE - Preformatted text
+
+TABLE - Table
+  output TRs as lines, csv the TDs
+
+UL - Unordered list
+
+DD - Definition description
+DT - Definition term
+LI - List item
+
+
+a (Links)     -   Link to Somewhere[1] ---- [1] http://
+b (Bold)      -   *bold*
+i (Italic)    -   _italic_
+strong        -   *strong*
+em (emphasis) -   _emphasis_
+u (underline) -   probably do nothing, maybe something like em

data/VERSION.yml

@@ -0,0 +1,4 @@
+--- 
+:patch: 9
+:major: 0
+:minor: 1

data/lib/mlist/email.rb

@@ -0,0 +1,69 @@
+module MList
+  
+  class Email < ActiveRecord::Base
+    set_table_name 'mlist_emails'
+    
+    include MList::Util::EmailHelpers
+    include MList::Util::TMailReaders
+    
+    def been_here?(list)
+      tmail.header_string('x-beenthere') == list.address
+    end
+    
+    def date
+      if date_from_email = super
+        return date_from_email
+      else
+        self.created_at ||= Time.now
+      end
+    end
+    
+    def from
+      tmail.header_string('from')
+    end
+    
+    # Answers the usable destination addresses of the email.
+    #
+    def list_addresses
+      bounce? ? tmail.header_string('to').match(/\Amlist-(.*)\Z/)[1] : recipient_addresses
+    end
+    
+    # Answers true if this email is a bounce.
+    #
+    # TODO Delegate to the email_server's bounce detector.
+    #
+    def bounce?
+      tmail.header_string('to') =~ /mlist-/
+    end
+    
+    def tmail=(tmail)
+      @tmail = tmail
+      write_attribute(:source, tmail.port.read_all)
+    end
+    
+    def tmail
+      @tmail ||= TMail::Mail.parse(source)
+    end
+    
+    # Provide reader delegation to *most* of the underlying TMail::Mail
+    # methods, excluding those overridden by this Class and the [] method (an
+    # ActiveRecord method).
+    def method_missing(symbol, *args, &block) # :nodoc:
+      if symbol.to_s !~ /=\Z/ && symbol != :[] && symbol != :source && tmail.respond_to?(symbol)
+        tmail.__send__(symbol, *args, &block)
+      else
+        super
+      end
+    end
+    
+    # Answers the set of addresses found in the TO and CC fields of the email.
+    #
+    def recipient_addresses
+      (Array(tmail.to) + Array(tmail.cc)).collect(&:downcase).uniq
+    end
+    
+    def respond_to?(method)
+      super || (method.to_s !~ /=\Z/ && tmail.respond_to?(method))
+    end
+  end
+end

data/lib/mlist/email_post.rb

@@ -0,0 +1,126 @@
+module MList
+  
+  # The simplest post that can be made to an MList::MailList. Every instance
+  # must have at least the text content and a subject. Html may also be added.
+  #
+  # It is important to understand that this class is intended to be used by
+  # applications that have some kind of UI for creating a post. It assumes
+  # Rails form builder support is desired, and that there is no need for
+  # manipulating the final TMail::Mail object that will be delivered to the
+  # list outside of the methods provided herein.
+  #
+  class EmailPost
+    include MList::Util::EmailHelpers
+    
+    ATTRIBUTE_NAMES = %w(copy_sender html text mailer subject subscriber)
+    ATTRIBUTE_NAMES.each do |attribute_name|
+      define_method(attribute_name) do
+        @attributes[attribute_name]
+      end
+      define_method("#{attribute_name}=") do |value|
+        @attributes[attribute_name] = value
+      end
+    end
+    
+    attr_reader :parent_identifier, :reply_to_message
+    
+    def initialize(attributes)
+      @attributes = {}
+      self.attributes = {
+        :mailer => 'MList Client Application'
+      }.merge(attributes)
+    end
+    
+    def attributes
+      @attributes.dup
+    end
+    
+    def attributes=(new_attributes)
+      return if new_attributes.nil?
+      attributes = new_attributes.dup
+      attributes.stringify_keys!
+      attributes.each do |attribute_name, value|
+        send("#{attribute_name}=", value)
+      end
+    end
+    
+    def copy_sender=(value)
+      @attributes['copy_sender'] = %w(true 1).include?(value.to_s)
+    end
+    
+    def reply_to_message=(message)
+      if message
+        @parent_identifier = message.identifier
+      else
+        @parent_identifier = nil
+      end
+      @reply_to_message = message
+    end
+    
+    def subject
+      @attributes['subject'] || (reply_to_message ? "Re: #{reply_to_message.subject}" : nil)
+    end
+    
+    def to_s
+      to_tmail.to_s
+    end
+    
+    def to_tmail
+      raise ActiveRecord::RecordInvalid.new(self) unless valid?
+      
+      builder = MList::Util::TMailBuilder.new(TMail::Mail.new)
+      
+      builder.mime_version = "1.0"
+      builder.mailer = mailer
+      
+      if parent_identifier
+        builder.in_reply_to = parent_identifier
+        builder.references = [bracket(parent_identifier)]
+      end
+      
+      builder.from = subscriber_name_and_address(subscriber)
+      builder.subject = subject
+      
+      if html
+        builder.add_text_part(text)
+        builder.add_html_part(html)
+        builder.set_content_type('multipart/alternative')
+      else
+        builder.body = text
+        builder.set_content_type('text/plain')
+      end
+      
+      builder.tmail
+    end
+    
+    # vvv  ActiveRecord validations interface implementation  vvv
+    
+    def self.human_name(options = {})
+      self.name.humanize
+    end
+    
+    def self.self_and_descendants_from_active_record #nodoc:
+      [self]
+    end
+    
+    def self.human_attribute_name(attribute_key_name, options = {})
+      attribute_key_name.humanize
+    end
+    
+    def errors
+      @errors ||= ActiveRecord::Errors.new(self)
+    end
+    
+    def validate
+      errors.clear
+      errors.add(:subject, 'required') if subject.blank?
+      errors.add(:text, 'required') if text.blank?
+      errors.add(:text, 'needs to be a bit longer') if !text.blank? && text.strip.size < 25
+      errors.empty?
+    end
+    
+    def valid?
+      validate
+    end
+  end
+end

data/lib/mlist/email_server/base.rb

@@ -0,0 +1,33 @@
+module MList
+  module EmailServer
+    class Base
+      attr_reader :settings
+      
+      def initialize(settings)
+        @settings = {
+          :domain => ::Socket.gethostname
+        }.merge(settings)
+        
+        @uuid = UUID.new
+        @receivers = []
+      end
+      
+      def deliver(tmail)
+        raise 'Implement actual delivery mechanism in subclasses'
+      end
+      
+      def generate_message_id
+        "#{@uuid.generate}@#{@settings[:domain]}"
+      end
+      
+      def receive(tmail)
+        email = MList::Email.new(:tmail => tmail)
+        @receivers.each { |r| r.receive_email(email) }
+      end
+      
+      def receiver(rx)
+        @receivers << rx
+      end
+    end
+  end
+end

data/lib/mlist/email_server/default.rb

@@ -0,0 +1,31 @@
+module MList
+  module EmailServer
+    
+    class Default < Base
+      def initialize(incoming_server, outgoing_server, settings = {})
+        super(settings)
+        @incoming_server, @outgoing_server = incoming_server, outgoing_server
+        @incoming_server.receiver(self)
+      end
+      
+      # Delegates delivery of email to outgoing server.
+      #
+      def deliver(tmail)
+        @outgoing_server.deliver(tmail)
+      end
+      
+      # Delegates fetching emails to incoming server.
+      def execute
+        @incoming_server.execute
+      end
+      
+      # Delegates processing of email from incoming server to receivers on
+      # self.
+      #
+      def receive_email(email)
+        @receivers.each { |r| r.receive_email(email) }
+      end
+    end
+    
+  end
+end

data/lib/mlist/email_server/fake.rb

@@ -0,0 +1,16 @@
+module MList
+  module EmailServer
+    class Fake < Base
+      attr_reader :deliveries
+      
+      def initialize(settings = {})
+        super
+        @deliveries = []
+      end
+      
+      def deliver(tmail)
+        @deliveries << tmail
+      end
+    end
+  end
+end

data/lib/mlist/email_server/pop.rb

@@ -0,0 +1,28 @@
+require 'pop_ssl'
+
+module MList
+  module EmailServer
+    
+    class Pop < Base
+      def deliver(tmail)
+        raise "Mail cannot be delivered through a POP server. Please use the '#{MList::EmailServer::Default.name}' type."
+      end
+      
+      def execute
+        connect_to_email_account do |pop|
+          pop.mails.each { |message| receive(TMail::Mail.parse(message.pop)); message.delete }
+        end
+      end
+      
+      private
+        def connect_to_email_account
+          pop3 = Net::POP3.new(settings[:server], settings[:port], false)
+          pop3.enable_ssl if settings[:ssl]
+          pop3.start(settings[:username], settings[:password]) do |pop|
+            yield pop
+          end
+        end
+    end
+    
+  end
+end

data/lib/mlist/email_server/smtp.rb

@@ -0,0 +1,24 @@
+require 'net/smtp'
+
+module MList
+  module EmailServer
+    
+    class Smtp < Base
+      def deliver(tmail)
+        destinations = tmail.destinations
+        tmail.delete_no_send_fields
+        smtp = Net::SMTP.new(settings[:address], settings[:port])
+        smtp.enable_starttls_auto if settings[:enable_starttls_auto] && smtp.respond_to?(:enable_starttls_auto)
+        smtp.start(settings[:domain], settings[:user_name], settings[:password],
+                   settings[:authentication]) do |smtp|
+          smtp.sendmail(tmail.encoded, tmail['sender'], destinations)
+        end
+      end
+      
+      def execute
+        raise "Mail cannot be received through an SMTP server. Please use the '#{MList::EmailServer::Default.name}' type."
+      end
+    end
+    
+  end
+end

data/lib/mlist/email_server.rb

@@ -0,0 +1,2 @@
+require 'mlist/email_server/base'
+require 'mlist/email_server/default'

data/lib/mlist/email_subscriber.rb

@@ -0,0 +1,6 @@
+module MList
+  
+  # Represents a simple subscriber instance, wrapping an email address.
+  #
+  EmailSubscriber = Struct.new('EmailSubscriber', :email_address)
+end