newman 0.1.1 → 0.2.0

data/CHANGELOG.md

@@ -0,0 +1,23 @@
+### 0.2.0 (2012-02-08)
+
+- Internals mostly rewritten, changes too numerous to outline meaningfully. We'll keep better track of these changes in the future.
+
+- Basic logging support added.
+
+- Lots of documentation added.
+
+- Server no longer crashes upon application errors unless
+  service.raise_exceptions is set to true.
+
+- Added Newman::Controller#skip_response which allows disabling delivery of a response
+  email upon demand.
+
+- Make Newman::MailingList fail gracefully by checking subscriber status before
+  attempting subscribe / unsubscribe.
+
+- Add a generic callback method to Newman::Application which allows for arbitrary
+  callbacks to be run based on filters against the Mail::Message object.
+
+### 0.1.1 (2012-02-03)
+
+- First official release.

data/Gemfile

@@ -0,0 +1,3 @@
+source :rubygems
+
+gemspec

data/README.md

@@ -1,10 +1,129 @@
-## Newman: Because the mail never stops!
+![Newman](http://i.imgur.com/92bZB.jpg)
 
-This is going to be an experiment in building a simple framework for developing
-email-based applications. It is based on ideas from an exercise in 
-Mendicant University's January 2012 core skills course, and also serves as a
-demo application for an article in the Practicing Ruby Journal.
+Newman is a microframework which aims to do for email-based 
+applications what Rack and Sinatra have done for web programming. **While our
+goals may be ambitious, this project is
+currently in a very early experimental stage, and is in no way safe for use in
+production.** 
 
-For a demonstration of how Newman is used, check out
-[Jester](http://github.com/mendicant-university/jester) as well as some of the
+That said, Newman is already capable of doing a number of interesting things. In
+particular:
+
+* A simple polling server provides a basic interface for
+  reading email from a single inbox and then building up a response email.
+
+* Filters are provided for handling emails based on their TO and SUBJECT fields.
+
+* Filters can also be defined for handling messages based on arbitrary
+  conditions evaluated against a `Mail::Messsage` object.
+
+* A rudimentary PStore backed storage mechanism is provided for persistence.
+
+* Basic support for maintaining persistent lists of email addresses is
+  provided.
+
+* Basic support for email templates are provided via Tilt.
+
+* A barebones configuration system allows configuring email settings, service
+  settings, and application specific settings.
+
+We are still working on figuring out what belongs in Newman and what doesn't,
+and so all of these features are subject to change or disappear entirely. But if
+you have a need for this sort of tool, it's worth noting that this software
+isn't entirely vaporware, and that we could use your help!
+
+### Scary Warning:
+
+DON'T HOOK UP NEWMAN TO ANY EMAIL INBOX THAT YOU CAN'T COMPLETELY WIPE OUT EVERY TIME IT RUNS. NEWMAN **WILL** DELETE YOUR EMAILS!!!
+
+### For a demonstration of how Newman is used:
+
+Check out [Jester](http://github.com/mendicant-university/jester) as well as some of the
 simple examples in this repository.
+
+### For a walkthrough of Newman's codebase:
+
+Check out [Newman's Rocco-based API documentation](http://mendicant-university.github.com/newman/lib/newman.html).
+
+### For general discussion, questions, and ideas about Newman:
+
+Find seacreature or ericgj in the #newman channel on Freenode or send an email to newman@librelist.org
+
+### Contributing to Newman:
+
+We do not yet have a clear roadmap or contributor guidelines, so be sure to talk
+to us before working on bug fixes or patches. But assuming you do want to send
+us some code, here is what you need to know:
+
+* You get to keep the copyright to your code, but you must agree to license it
+  under the MIT license.
+
+* Your code should come with tests. Integration tests are fine, but unit tests
+  would be nice where appropriate. Right now Newman is under tested and we don't
+  want to make that problem worse. You can of course submit a pull request for
+  feedback BEFORE writing tests.
+
+* Your code should be fully documented, and properly formatted for use with
+  Rocco. Please try to emulate the style and conventions we've been using where
+  possible. Do the best you can with this, and we'll help tighten up wording and
+  clean up formatting as needed. You can of course submit a pull request for
+  feedback BEFORE writing documentation.
+
+* Newman is taking a use-case oriented approach to design. Be prepared to
+  justify any proposed change with real or realistic scenarios, rather than
+  simply addressing theoretical concerns.
+
+### Versioning Policy:
+
+We will try to follow the guidelines below when cutting new releases,
+to the extent that it makes sense to do so.
+
+1) Clearly mark each object that Newman provides as being part of either
+the 'external API' or the 'internal API'. The external API is for
+application developers, the internal API is for Newman itself as well as
+extension developers
+
+2) Before 1.0, allow backwards incompatible internal API changes during
+every release, and allow backwards incompatible external API changes
+during minor version bumps, but do not add or change external behavior
+in tiny version bumps.
+
+3) After 1.0, do not allow external or internal API changes during tiny
+version bumps (these will be bug fixes only). Allow changes to the
+internal API during minor version bumps, but maintain backwards
+compatibility with the 1.0 release (i.e. things introduced after 1.0 can
+be changed / removed, but things which shipped with 1.0 should stay
+supported, even internally). Allow external API changes or
+backwards-incompatible changes to the internals only on major version
+bumps (i.e. 2.0, 3.0, etc). Use semantic versioning and declare the
+external API to be the 'public' API.
+
+We plan to get to 1.0 quickly to reach a stabilizing point for application
+developers and a slower moving target for extension developers. This means
+that our 1.0 release will be more of a minimum-viable product and not
+necessarily a full-stack framework.
+
+### Authorship:
+
+Newman is being developed by [Gregory Brown](http://community.mendicantuniversity.org/people/sandal)
+and [Eric Gjertsen](http://community.mendicantuniversity.org/people/ericgj), along with 
+help from several other folks. 
+
+It is based on an assignment from [Mendicant
+University](http://mendicantuniversity.org)'s January 2011 core
+skills course, and was also used as a sample application for the [Practicing Ruby](http://practicingruby.com)
+journal. The original inspiration for this project came from some code
+written by [Brent Vatne](http://community.mendicantuniversity.org/people/brentvatne), 
+as well as from the general ideas behind Rack and Sinatra.
+
+[View the full list of contributors](https://github.com/mendicant-university/newman/contributors) to see who else has helped out.
+
+### License:
+
+Copyright (c) 2012 Gregory Brown, Eric Gjertsen, et al.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/examples/config/dreamhost.rb.example

@@ -1,9 +1,13 @@
+# WARNING: DON'T HOOK UP NEWMAN TO ANY EMAIL INBOX THAT
+# YOU CAN'T COMPLETELY WIPE OUT EVERY TIME IT RUNS.
+# NEWMAN **WILL** DELETE YOUR EMAILS!!!
+
 imap.address   = "mail.<<DOMAIN>>"
 imap.user      = "<<USERNAME>>@<<DOMAIN>>"
 imap.password  = "<<PASSWORD>>"
 
 smtp.address   = "mail.<<DOMAIN>>"
-smtp.user      = "<<USER>>@<<DOMAIN>>"
+smtp.user      = "<<USERNAME>>@<<DOMAIN>>"
 smtp.password  = "<<PASSWORD>>" 
 
 service.domain           = "<<DOMAIN>>"

data/examples/config/gmail.rb.example

@@ -1,3 +1,7 @@
+# WARNING: DON'T HOOK UP NEWMAN TO ANY EMAIL INBOX THAT
+# YOU CAN'T COMPLETELY WIPE OUT EVERY TIME IT RUNS.
+# NEWMAN **WILL** DELETE YOUR EMAILS!!!
+
 imap.address     = "imap.gmail.com"
 imap.user        = "<<USERNAME>>@gmail.com"
 imap.password    = "<<PASSWORD>>"

data/examples/live_test.rb

@@ -2,13 +2,10 @@ require_relative "ping_pong"
 
 
 settings = Newman::Settings.from_file("config/environment.rb")
+mailer = Newman::Mailer.new(settings)
 
-mailer = Newman::Mailer
-mailer.configure(settings)
-
-server = Newman::Server
-server.settings = settings
-server.mailer   = mailer
+server = Newman::Server.new(settings, mailer)
+server.apps << Newman::Examples::PingPong
 
 mailer.deliver_message(:to   => settings.application.ping_email,
                        :from => settings.service.default_sender)
@@ -20,7 +17,7 @@ settings.application.live_test_delay.downto(1) do |i|
 end
 puts
 
-Newman::Server.tick(Newman::Examples::PingPong)
+server.tick
 
 remaining_attempts = 3
 loop do

data/examples/simple_mailing_list.rb

@@ -12,7 +12,7 @@ module Newman
         end
       end
 
-      to(:tag, "subscribe") do
+      subject(:match, "subscribe") do
         if list.subscriber?(sender)
           respond :subject => "ERROR: Already subscribed",
                   :body    => template("subscribe-error")
@@ -24,7 +24,7 @@ module Newman
         end
       end
 
-      to(:tag, "unsubscribe") do
+      subject(:match, "unsubscribe") do
         if list.subscriber?(sender)
           list.unsubscribe(sender)
 

data/lib/newman.rb

@@ -1,3 +1,95 @@
+# [Newman](https://github.com/mendicant-university/newman) is a 
+# microframework which aims to do for email-based applications what Rack and 
+# Sinatra have done for web programming. **While our goals may be ambitious, 
+# this project is currently in a very early experimental stage, 
+# and is in no way safe for use in production.**
+#
+# That said, we still welcome contributors to help us work on this project
+# as we collectively figure out what exactly Newman should be. Even in its
+# very early stages, Newman is already doing some useful things and provides
+# a wide range of application development features via its external interface
+# as well as extension points via its internal interface.
+#
+# The documentation you're currently reading is meant to help explain Newman's
+# implementation to contributors and alpha testers. Before you dig deeper into
+# the source, make sure to read 
+# [Newman's README](https://github.com/mendicant-university/newman)
+# as well as the [Jester](https://github.com/mendicant-university/jester) demo
+# application.
+#
+# Assuming you have done those things and are now familiar with the basic ideas
+# behind Newman, the following outline may help you in your explorations of
+# its source code.
+#
+# ### External interface (for application developers)
+#
+# * [Newman::Server](http://mendicant-university.github.com/newman/lib/newman/server.html) 
+#   takes incoming mesages from a mailer object and passes them to applications as a 
+#   request, and then delivers a response email that built up by its
+#   applications.
+#
+# * [Newman::Application](http://mendicant-university.github.com/newman/lib/newman/application.html)
+#   provides the main entry point for Newman application developers, and exists to tie together 
+#   various Newman objects in a convenient way.
+#
+# * [Newman::Filters](http://mendicant-university.github.com/newman/lib/newman/filters.html)
+#   provides high level filters for matching incoming requests.
+#   
+# * [Newman::Controller](http://mendicant-university.github.com/newman/lib/newman/controller.html)
+#   provides a context for application callbacks to run in, and provides most of
+#   the core functionality for preparing a response email.
+#
+# * [Newman::MailingList](http://mendicant-university.github.com/newman/lib/newman/mailing_list.html)
+#   implements a simple mechanism for storing persistent lists of email addresses keyed 
+#   by a mailing list name. 
+#
+# * [Newman::Store](http://mendicant-university.github.com/newman/lib/newman/store.html) provides 
+#   a minimal persistence layer for storing non-relational data.
+#
+# * [Newman::Recorder](http://mendicant-university.github.com/newman/lib/newman/recorder.html)
+#   provides a mechanism for storing records with autoincrementing identifiers
+#   within a `Newman::Store` and supports some rudimentary CRUD functionality.
+#
+# * Implicitly, the settings files used by Newman as well as the structure of
+#   its low level data storage format are also considered part of external API,
+#   even though these are actually implementation details. This is simply
+#   because we want to make sure to clearly reflect backwards incompatible
+#   changes to these features via our versioning policy, as this sort of
+#   change could potentially cause update problems for application
+#   developers.
+#
+# ### Internal interface (for extension developers)
+#
+# * [Newman::EmailLogger](http://mendicant-university.github.com/newman/lib/newman/email_logger.html)
+#   provides rudimentary logging support for email objects, and primarily exists to 
+#   support the `Newman::RequestLogger` and `Newman::ResponseLogger` objects.
+#
+# * [Newman::RequestLogger](http://mendicant-university.github.com/newman/lib/newman/request_logger.html)
+#   provides a mechanism for logging information about incoming emails.
+#
+# * [Newman::ResponseLogger](http://mendicant-university.github.com/newman/lib/newman/response_logger.html)
+#   provides a mechanism for logging information about outgoing emails.
+#
+# * [Newman::Mailer](http://mendicant-university.github.com/newman/lib/newman/mailer.html) provides a thin
+#   wrapper on top of the [mail gem](http://github.com/mikel/mail) which is
+#   designed to have a minimal API so that it can easily be swapped out with
+#   another mailer object.
+#
+# * [Newman::TestMailer](http://mendicant-university.github.com/newman/lib/newman/test_mailer.html) 
+#   is a drop-in replacement for `Newman::Mailer` meant for use in automated testing. 
+#
+# * [Newman::Settings](http://mendicant-university.github.com/newman/lib/newman/settings.html) provides
+#   the base functionality that is used by Newman's configuration files. Note
+#   that while this object is part of the internals, the settings actually used
+#   by Newman should be considered part of the external API.
+#
+# ### Getting help, or helping out:
+#
+# Please catch up with seacreature or ericgj in the #newman channel on Freenode,
+# or send an email to newman@librelist.org. We'd love to hear any questions,
+# ideas, or suggestions you'd like to share with us.
+
+require "logger"
 require "pstore"
 require "ostruct"
 require "fileutils"
@@ -5,13 +97,17 @@ require "fileutils"
 require "mail"
 require "tilt"
 
+require_relative "newman/email_logger"
+require_relative "newman/request_logger"
+require_relative "newman/response_logger"
 require_relative "newman/server"
-require_relative "newman/commands"
+require_relative "newman/filters"
 require_relative "newman/application"
 require_relative "newman/controller"
 require_relative "newman/mailing_list"
 require_relative "newman/settings"
 require_relative "newman/store"
+require_relative "newman/recorder"
 require_relative "newman/mailer"
 require_relative "newman/test_mailer"
 require_relative "newman/version"

data/lib/newman/application.rb

@@ -1,6 +1,33 @@
+# `Newman::Application` provides the main entry point for Newman application
+# developers, and exists to tie together various Newman objects in a convenient
+# way.
+#
+# For an fairly complete example of a `Newman::Application` object in use, be
+# sure to check out [Jester](https://github.com/mendicant-university/jester).
+#
+# `Newman::Application` is part of Newman's **external interface**.
+
 module Newman 
   class Application
-    include Commands
+
+    include Filters
+    # ---
+    
+    # A `Newman::Application` object is a blank slate upon creation, with fields
+    # set to hold `callbacks`, `matchers`, and `extensions`. A block may
+    # optionally be provided, which then gets executed within the context of the
+    # newly created `Newman::Application` instance. This is the common way of
+    # building applications, and is demonstrated by the example below:
+    #  
+    #     ping_pong = Newman::Application.new do
+    #       subject("ping") do
+    #         respond(:subject => "pong")
+    #       end
+    #     end
+    # 
+    # Any method that can be called on a `Newman::Application` instance can be
+    # called within the provided block, including those methods mixed in by
+    # `Newman::Filters`.
 
     def initialize(&block)
       self.callbacks  = []
@@ -9,6 +36,18 @@ module Newman
 
       instance_eval(&block) if block_given?
     end
+    
+    # ---
+    
+    # `Newman::Application#call` accepts a hash of parameters which gets used to
+    # create a new `Newman::Controller` object. The controller is then extended
+    # by all of the modules stored in the `extensions` field on the application
+    # object, and is finally passed along to
+    # `Newman::Application#trigger_callbacks`, which does the
+    # magical work of figuring out which callbacks to run, if any.
+    #
+    # This method is meant to be run by a `Newman::Server` object, and isn't
+    # especially useful on its own.
 
     def call(params)
       controller = Controller.new(params)      
@@ -16,8 +55,206 @@ module Newman
       trigger_callbacks(controller)
     end
 
+    # ---
+   
+    # `Newman::Application#default` is used to define a default callback
+    # which will run when no other callbacks match the incoming request.
+    # For example, you can define a callback such as the one below:
+    #
+    #     default do
+    #       respond(:subject => "REQUEST NOT UNDERSTOOD")
+    #     end
+    #
+    # Unless you are building an application that will never fail to 
+    # match at least one of its filters, you MUST set up a default callback 
+    # if you want to avoid a possible application error. We know this is
+    # not exactly the most desireable behavior, and will try to fix this
+    # in a future version of Newman.
+
+    def default(&callback)
+      self.default_callback = callback
+    end
+
+    # ---
+    
+    # `Newman:::Application#use` is used to register the extension 
+    # modules that get mixed in to the controller 
+    # objects created by `Newman::Application#call`. This allows 
+    # an application object to provide extensions for use within its 
+    # callbacks, as in the example shown below.
+    #
+    #     module ListLoader
+    #       def load_list(name)
+    #         store = Newman::Store.new(settings.application.list_db)
+    #         Newman::MailingList.new(name, store)
+    #       end
+    #     end
+    #
+    #     list_app = Newman::Application.new do
+    #       use ListLoader
+    #       
+    #       match :list_id, "[^.]+"
+    #
+    #       to(:tag, "{list_id}.subscribe") do
+    #         list = load_list(params[:list_id])
+    #
+    #         if list.subscriber?(sender)
+    #           # send a failure message
+    #         else
+    #           # susbcribe the user and send a welcome message
+    #         end
+    #       end
+    #     end
+    #
+    # This method is mainly meant to be used with pre-packaged extensions or
+    # more complicated forms of callback helpers. This example was just shown
+    # for the sake of its simplicity, but for similar use cases it would
+    # actually be better to use `Newman::Application#helpers`
+
+    def use(extension)
+      extensions << extension
+    end
+
+    # ---
+    
+    # `Newman::Application#helpers` is used to build simple controller
+    # extensions, and is mostly just syntactic sugar. For example, 
+    # rather than using an explicit
+    # module, the example shown in the `Newman::Application#use` documentation
+    # can be rewritten as follows:
+    #
+    #     list_app = Newman::Application.new do
+    #       helpers do
+    #         def load_list(name)
+    #           store = Newman::Store.new(settings.application.list_db)
+    #           Newman::MailingList.new(name, store)
+    #         end
+    #       end
+    #       
+    #       match :list_id, "[^.]+"
+    #
+    #       to(:tag, "{list_id}.subscribe") do
+    #         list = load_list(params[:list_id])
+    #
+    #         if list.subscriber?(sender)
+    #           # send a failure message
+    #         else
+    #           # susbcribe the user and send a welcome message
+    #         end
+    #       end
+    #     end
+    #
+    # It's important to note that for any controller extensions that might be
+    # reusable, or for more complicated logic, `Newman::Application#use` is
+    # probably a better tool to use.
+
+    def helpers(&block)
+      use Module.new(&block)
+    end
+
+    # ---
+    
+    # `Newman::Application#match` is used to define patterns which are used for
+    # extracting callback parameters. An example is shown below:
+    #
+    #     jester = Newman::Application.new do
+    #       match :genre, '\S+'
+    #       match :title, '.*'
+    #
+    #       subject(:match, "a {genre} story '{title}'") do
+    #         story_library.add_story(:genre => params[:genre],
+    #                                 :title => params[:title],
+    #                                 :body => request.body.to_s)
+    #
+    #         respond :subject => "Jester saved '#{params[:title]}'"
+    #       end
+    #     end
+    #
+    # Because Newman's built in filters are designed to escape regular
+    # expression syntax by default, `Newman::Application#match` provides the
+    # only high-level mechanism for dynamic filter matches. Low level matching
+    # is possible via `Newman::Application#callback`, but would
+    # be an exercise in tedium for most application developers.
+
+    def match(name, pattern)
+      matchers[name] = pattern
+    end
+
+    # ---
+    
+    # `Newman::Application#callback` is a low level feature for defining custom
+    # callbacks. For ideas on how to roll your own filters with it, see the
+    # implementation of the `Newman::Filters` module.
+
+    def callback(action, filter)
+      callbacks << { :filter   => filter, 
+                     :action   => action }
+    end
+
+    # ---
+    
+    # `Newman::Application#compile_regex` is used for converting pattern strings
+    # into a regular expression string suitable for use in filters. This method is a
+    # low-level feature, and is not meant for use by application developers. See
+    # the `Newman::Filters` module for how to use it to build your own callback
+    # filters.
+
+    def compile_regex(pattern)
+      Regexp.escape(pattern)
+            .gsub(/\\{(.*?)\\}/) { |m| "(?<#{$1}>#{matchers[$1]})" } 
+    end
+
+    # ---
+    
+    # **NOTE: Methods below this point in the file are implementation 
+    # details, and should not be depended upon.**
+
     private
 
+    # ---
+    
+    # `Newman::Application#trigger_callbacks` runs a two step process:
+    # 
+    # 1) It grabs the `filter` Proc for each callback and executes it, passing in
+    # the provided `controller`. Any `filter` proc that returns a logically true
+    # value is selected to be run.
+    #
+    # 2) If the selection of `matched_callbacks` is empty, it executes the default
+    # callback in the context of a controller object. Otherwise, it runs each 
+    # callback in sequence, in the context of a controller object.
+    #
+    # This method may have some weird side effects because it relies on
+    # awkward state mutations that could be either done in a better way or 
+    # replaced with a mostly stateless approach. We will look at fixing 
+    # this in a future Newman release.
+
+    def trigger_callbacks(controller)
+      matched_callbacks = callbacks.select do |e| 
+        filter = e[:filter]
+        e[:match_data] = filter.call(controller)
+      end
+
+      if matched_callbacks.empty?
+        controller.instance_exec(&default_callback) 
+      else
+        matched_callbacks.each do |e|
+          action = e[:action]
+          controller.params = e[:match_data] || {}
+          controller.instance_exec(&action)
+        end
+      end
+    end
+
+    # ---
+    
+    # These accessors have been made private to reflect the fact that
+    # `Newman::Application` is meant to be customized via its high level methods
+    # such as `use`, `helpers`, `match`, and `callback`. Any extensions of
+    # `Newman::Application` should rely on those methods and not directly
+    # reference these fields at all. If there is some functionality missing
+    # that needs to be added to the public API for you to build your
+    # extension, just let us know.
+
     attr_accessor :callbacks, :default_callback, :matchers, :extensions
   end
 end