watcher 1.1.0 → 1.2.0

data/History.txt

@@ -1,3 +1,60 @@
+== 1.2.0 2008-07-30
+
+* 6 major enhancements:
+  * Major re-write of entire Watcher code, separating logic into two class
+    files: watcher.rb and event.rb.
+  * Watcher verbosity levels consolidated to three levels: debug, verbose, and
+    always, in order from lowest priority to highest priority.  Debug events
+    will rarely be logged, while always events will always be logged.
+  * The Watcher instantiation parameter, :fail_actions Array, is unsupported
+    and replaced by a Hash named :default_fail.  Like :fail_actions, this
+    represents the set of actions to occur if no other fail actions are
+    specified when invoking one of Watcher's log event methods.  Unlike the
+    :fail_actions Array, this is a Hash, whose possible key/value pairs are
+    described below.
+  * Fail actions logic has been completely rewritten.
+    * Invocation of Watcher.debug, Watcher.verbose, and Watcher.always
+      requires a fail_action parameter.  This Hash must contain a :failure
+      key, equal to either :error or :warn.
+      * If :warn, then an exception will merely cause a warning event to
+        be recorded.
+      * If :error, an exception will cause an error event to be recorded, then
+        re-raise the original exception.
+    * Two other keys are permitted in the fail_action parameter.  The presence
+      of one necessitates the presence of the other.
+      * :tries -- the total number of tries Watcher will attempt to execute
+        the code in the block.
+      * :proc  -- the Proc instance that Watcher will invoke before each
+        retry attempt.
+      * If neither :tries nor :proc is given, Watcher takes action as if
+        :tries was set to 1 and :proc was set to lambda { }.  However, Watcher
+        will throw an ArgumentError exception if :tries is specified, and not
+        a Fixnum with a value of at least 2.
+      * NB:  The Proc will be invokes no more than (:tries - 1) times.  This
+        is very different than Watcher 1.1.0, where the key was :retry, and
+        whose value determined the number of retries that would occur.  In
+        the previous version, the number of times the Proc would be invoked
+        was equal to the :retry value.  This is no longer the case.
+  * Indentation logic re-written.
+    * Log entries suppressed due to Watcher's verbosity setting are queued up.
+      If an exception is thrown, Watcher will replay those previously
+      suppressed log entries to allow a more detailed analysis of the cause of
+      the exception.
+    * Format of log output is nested in a hierarchical fashion, allowing
+      user to more easily trace the progression of tasks when trouble-shooting
+      program output via its logs.
+  * Unit tests written, although perhaps not yet entirely comprehensive.
+
+* 3 minor enhancements:
+  * Allows either append, or overwrite mode for logs.  Watcher can be
+    created with the :write parameter set to either :append or :overwrite.
+    The default is :overwrite.
+  * Watcher will ensure output stream is an actual file before it attempts to
+    send it via email.  If the output stream is not a file, then it logs an
+    error message to the output stream.
+  * Eliminated use of continuations in preference to using the standard retry
+    statement.
+
 == 1.1.0 2008-07-22
 
 * 1 major enhancement:

data/Manifest.txt

@@ -4,9 +4,10 @@ Manifest.txt
 PostInstall.txt
 README.txt
 Rakefile
-TODO
+TODO.txt
 config/hoe.rb
 config/requirements.rb
+lib/event.rb
 lib/watcher.rb
 lib/watcher/version.rb
 script/console
@@ -17,7 +18,10 @@ setup.rb
 tasks/deployment.rake
 tasks/environment.rake
 tasks/website.rake
-test/watcher-example.rb
+test/tc_event.rb
+test/test_helper.rb
+test/test_watcher.rb
+test/ts_package.rb
 website/index.html
 website/index.txt
 website/javascripts/rounded_corners_lite.inc.js

data/README.txt

@@ -33,42 +33,44 @@ for the various actions taken.
 
 * Exception handling without the boiler-plate code.
 * Multiple tiers of verbosity.
-* Flexible logging capabilities, including to files, STDERR, email.
+* Flexible logging capabilities, including to files, STDERR, and email.
 * Nested, and hierarchical log output for descriptive stack-frame level
   understanding where an exception was thrown.
 * Ability to change the time format of your logs by supplying an arbitrary
   Proc to return the time in the format of your choice.
-* Ability to customize the layout of the logs, by turning off hierarchical
-  view, or changing Strings used to indicate events.
+* Ability to customize the layout of the logs.
 * Email log files automatically, based on number of errors or warnings.
 * Ability to invoke arbitrary sequence of events if exception is thrown,
-  including invocation of Proc objects.
+  by means of invocation of Proc object.
 
 == SYNOPSIS:
 
 # create Watcher instance with slightly modified defaults
-$watcher=Watcher.create(:output=>log_file, :verbosity=>:warn)
+$watcher = Watcher.create(:output => log_file)
 
-$watcher.info("starting #{File.basename($0)}") do
+$watcher.debug("starting #{File.basename($0)}") do
   # Look for CONFIG_PATH.  If not found, log warning then re-write it
-  fail_actions=[:log, lambda {write_config(config_path)}]
-  $watcher.info("searching for [#{config_path}].", fail_actions) do
-    if not File.exists?(config_path)
-      raise RuntimeError.new("WARNING: Missing [#{config_path}].  Creating a default config file.  (You should probably verify it...)")
-    end
+  params = {:tries => 2, :failure => :error,
+            :proc => lambda { |e| write_config(config_path) }}
+  msg = "ensuring existance of config file [#{config_path}]"
+  $watcher.verbose(msg, params) do
+    raise "file not found" unless File.file?(File.expand_path(config_path))
   end
 
   # invoke utility, which uses $watcher as well
-  fsc=FileSystemChecker.new(:config_path=>config_path)
-  fsc.check_file_systems(:all=>all_flag, :remount=>remount_flag)
-  if email_flag and $watcher.errors > 0
-    $watcher.send_email(email,'FAIL: Required filesystems are not mounted')
-  end
+  fsc = FileSystemChecker.new(:config_path => config_path)
+  fsc.check_file_systems(:all => all_flag, :remount => remount_flag)
+end
+
+if email_flag and ($watcher.errors > 0 or $watcher.warnings > 0)
+  $watcher.send_email(email, 'FAIL: Required filesystems are not mounted')
 end
 
 == REQUIREMENTS:
 
-* Passion for coding.
+* hoe
+
+* newgem
 
 == INSTALL:
 

data/TODO.txt

@@ -0,0 +1,33 @@
+1.  CRITICAL
+
+    a.  None.
+
+2.  MINOR
+
+    a.  Update source comments in watcher.rb and event.rb
+
+    b.  Ensure 78 character max in source files.
+
+    c.  Write some non-trivial example code.  I have developed and ported
+        several non-trivial utilities from Logger to Watcher, and am extremely
+        happy with the results.  Now let me get something ready to package
+        with this gem.
+
+    d.  Trap signals to clean up properly.
+
+    e.  Log rotation.
+
+3.  LOW
+
+    a.  Figure out how to get ri documentation working.
+
+    b.  Provide a Watcher.destroy method which closes the Watcher instance and
+        changes the @@watcher class attribute back to nil.
+
+    c.  Validate data when attributes accessors invoked.
+
+    d.  Use a Ruby MTA, as opposed to the UNIX mail program.
+
+    e.  As populate Event instance, remove k,v from Hash as assign, then
+        ensure fail_actions is empty to validate no extra invalid options
+        sent.

data/config/hoe.rb

@@ -1,7 +1,7 @@
 require 'watcher/version'
 
 AUTHOR = 'Karrick McDermott'  # can also be an array of Authors
-EMAIL = "karrick@karrick.net"
+EMAIL = "karrick@karrick.org"
 DESCRIPTION = "Watcher provides advanced integrated exception handling and logging functionality to your Ruby programs."
 GEM_NAME = 'watcher' # what ppl will type to install your gem
 RUBYFORGE_PROJECT = 'watcher' # The unix name for your project
@@ -13,7 +13,7 @@ EXTRA_DEPENDENCIES = [
 
 @config_file = "~/.rubyforge/user-config.yml"
 @config = nil
-RUBYFORGE_USERNAME = "unknown" # Don't change this.  Modify your "~/.rubyforge/user-config.yml" file.
+RUBYFORGE_USERNAME = "unknown"
 def rubyforge_username
   unless @config
     begin
@@ -55,21 +55,21 @@ $hoe = Hoe.new(GEM_NAME, VERS) do |p|
   p.description = DESCRIPTION
   p.summary = DESCRIPTION
   p.url = HOMEPATH
-  p.remote_rdoc_dir = '' # Release to root (Watcher is my only project)
-  p.rsync_args << ' --exclude=statsvn/'
   p.rubyforge_name = RUBYFORGE_PROJECT if RUBYFORGE_PROJECT
   p.test_globs = ["test/**/test_*.rb"]
   p.clean_globs |= ['**/.*.sw?', '*.gem', '.config', '**/.DS_Store']  #An array of file patterns to delete on clean.
+#  p.remote_rdoc_dir = '' # Release to root (Watcher is my only project)
+#  p.rsync_args << ' --exclude=statsvn/'
 
   # == Optional
   p.changes = p.paragraphs_of("History.txt", 0..1).join("\n\n")
   #p.extra_deps = EXTRA_DEPENDENCIES
 
-  #p.spec_extras = {}    # A hash of extra values to set in the gemspec.
-end
+    #p.spec_extras = {}    # A hash of extra values to set in the gemspec.
+  end
 
 CHANGES = $hoe.paragraphs_of('History.txt', 0..1).join("\\n\\n")
 PATH    = (RUBYFORGE_PROJECT == GEM_NAME) ? RUBYFORGE_PROJECT : "#{RUBYFORGE_PROJECT}/#{GEM_NAME}"
 $hoe.remote_rdoc_dir = File.join(PATH.gsub(/^#{RUBYFORGE_PROJECT}\/?/,''), 'rdoc')
 $hoe.rsync_args = '-av --delete --ignore-errors'
-$hoe.spec.post_install_message = File.open(File.dirname(__FILE__) + "/../PostInstall.txt").read rescue ""
+$hoe.spec.post_install_message = File.open(File.dirname(__FILE__) + "/../PostInstall.txt").read rescue ""

data/lib/event.rb

@@ -0,0 +1,177 @@
+#!/usr/bin/ruby -w
+# -*- Ruby -*-
+# Event
+#
+# Summary::   Event instance handles exception recovery, proper hierarchical
+#             format String for the log based on its level in the call-stack.
+# Author::    Karrick McDermott (karrick@karrick.org)
+# Date::      2008-07-22
+# Copyright:: Copyright (c) 2008 by Karrick McDermott.  All rights reserved.
+# License::   Simplified BSD License.
+
+######################################################################
+
+# Add this directory to front of require path.
+$:.unshift(File.dirname(__FILE__)) unless
+  $:.include?(File.dirname(__FILE__)) || $:.include?(File.expand_path(File.dirname(__FILE__)))
+
+######################################################################
+# An Event instance tracks the title of the task, and the required indentation
+# to display the task in the log.  This is required to store an Array of
+# events that took place--but below the verbosity threshhold--to replay them
+# into the log at the appropriate indention level should an exception be
+# thrown.
+class Event
+  ##### Publically read-write attributes
+
+  ##### Publically read-only attributes
+  attr_reader :failure      # what to do if all tries fail (:warn | :error)
+  attr_reader :hier         # shows nested levels
+  attr_reader :proc         # Proc to call before a retry
+  attr_reader :tries        # number of tries remaining
+  attr_reader :title        # event title
+
+  # Get an Event which is the child of this event
+  def child
+    self.dup.child!
+  end
+  
+  # Make this event into its child
+  def child!
+    if @hier.count('.') % 2 == 0 # if even number of periods
+      @hier << '.a'
+    else
+      @hier << '.0'
+    end
+    self
+  end
+
+  # Get copy of Event hier to the next task at the same level of the call-stack
+  def sibling
+    self.dup.sibling!
+  end
+
+  # Take this Event hier to the next task at the same level of the call-stack
+  def sibling!
+    # Now change the last hier to its successor
+    @hier.sub!(/[^.]+$/) { $&.succ }
+    self
+  end
+  
+  ########## Return String ready for log
+  def to_s
+    @time + ': ' + @hier + ' ### ' + @title
+  end
+  
+  ########################################
+  private
+  ########################################
+  
+  # Create a new Event instance with the specified task title, and an optional
+  # prefix for parent event id (used for log replay)
+  def initialize(time, hier, title, fail_actions, relationship)
+    @hier = hier
+    @time = time
+    @title = title
+
+    # validate title
+    unless title.respond_to? :to_s
+      # Defensive programming against Watcher's caller:
+      msg = "Event.initialize: The first argument (title) must respond to" \
+        + " the :to_s method. Your argument class is #{title.class}." \
+        + " Aborting task."
+      raise ArgumentError.new(msg)
+    end
+
+    # validate time
+    unless time.respond_to? :to_s
+      # Defensive programming against Watcher's caller:
+      msg = "Event.initialize: The second argument (time) must respond to" \
+        + " the :to_s method. Your argument class is #{time.class}." \
+        + " Aborting [#{title}]."
+      raise ArgumentError.new(msg)
+    end
+
+    # validate fail actions
+    if fail_actions.kind_of? Hash
+      # validate fail_actions (throws ArgumentError if bad; let caller catch)
+      @proc = fail_actions[:proc]
+      @tries = fail_actions[:tries]
+      @failure = fail_actions[:failure]
+      validate_fail_actions
+    else
+      # Defensive programming against Watcher's caller:
+      msg = "Event.initialize: The fourth argument (fail_actions) must be" \
+        + " a Hash, not a #{fail_actions.class}. Aborting [#{title}]."
+      raise ArgumentError.new(msg)
+    end
+    
+    # validate hier argument
+    if hier == nil
+      # If no hier event, then this Event has no parent or siblings.
+      @hier = '0'
+    elsif not hier.kind_of? String
+      # Defensive programming against Watcher.monitor
+      msg = "Event.initialize: sixth argument (hier) must be either nil," \
+        + " or of non-zero length String, not a #{hier.class}. Aborting [#{title}]."
+      raise ArgumentError.new(msg)
+    elsif hier == ''
+      # Defensive programming against Watcher.monitor
+      msg = "Event.initialize: sixth argument (hier) must be either nil," \
+        + " or of non-zero length String. Aborting [#{title}]."
+      raise ArgumentError.new(msg)
+    else # this event is related to hier
+      case relationship
+      when :child
+        @hier = hier
+        child!
+      when :sibling
+        @hier = hier
+        sibling!
+      else
+        msg = "Event.initialize: fifth argument (relationship) must be one" \
+          + " of either :child or :sibling"
+        raise ArgumentError.new(msg)
+      end
+    end
+  end
+
+# Deprecated, but worked fine at the time it was taken out.
+# # Take this Event hier to back up one hier from the call-stack
+# def dec
+#   # knock off the last segment
+#   @hier.sub!(/\.?[^.]+$/, '')
+#   # call sibling to increment parent hier
+#   sibling!
+# end
+
+  # Validate fail_actions parameter (throws ArgumentError if bad)
+  # * Must include either :warn or :error, but not both in Hash.
+  # * If :tries is specified, it must be a Fixnum, and then :proc must also
+  #   be specified, and a valid Proc instance.
+  # * If :tries is not specified, then :proc may not be specified.
+  def validate_fail_actions
+    # Either :warn or :error, not both, and not neither.
+    if @failure != :warn and @failure != :error
+      msg = "Event.validate_fail_actions: Must specify either :warn or :error"
+      raise ArgumentError.new(msg)
+    end
+
+    # If :tries is set, then it must be a Fixnum
+    if @tries != nil
+      if @tries.class != Fixnum or @tries < 2
+        raise ArgumentError.new(':tries must be at least 2 in order to specify a Proc')
+      else # must have a valid Proc
+        if @proc.class != Proc
+          raise ArgumentError.new('must specify a Proc instance when :tries is set')
+        end
+      end
+    else # tries is nil
+      # Ensure :proc not set
+      if @proc != nil
+        raise ArgumentError.new(':tries must be at least 2 in order to specify a Proc')
+      end
+      @tries = 1 # just one try since :proc was nil
+    end
+  end
+end

data/lib/watcher.rb

@@ -9,50 +9,46 @@
 # Copyright:: Copyright (c) 2008 by Karrick McDermott.  All rights reserved.
 # License::   Simplified BSD License.
 
+######################################################################
+
+# add this directory to front of require path
+$:.unshift(File.dirname(__FILE__)) unless
+  $:.include?(File.dirname(__FILE__)) || $:.include?(File.expand_path(File.dirname(__FILE__)))
+
+# Co-located in same directory as watcher.
+require 'event'
+
 ######################################################################
 class Watcher
-  VERSION = '1.1.0'
+  # Array of actions to perform when an Exception is thrown by code in the
+  # yield block.
+  # * The Watcher instance maintains a default @default_fail Hash for use when
+  #   a given invocation does not specify a different set of actions to
+  #   perform.
+  # * The default value is a Hash with one key, the value of which causes
+  #   Watcher to re-raise the exception if one should occur.
+  attr_accessor :default_fail
+  
+  # A String to include inside the log entry when an Exception is thrown.
+  # * This may be required in some circumstances where a separate log
+  #   monitoring program is scanning log files looking for a string such as,
+  #   ERROR or FAILURE.
+  # * The default value is nil.
+  attr_accessor :error_symbol
+
   # Tracks the number of times an Exception has invoked the :raise fail
   # action.
   # * Fail actions to log an error (:log), invoke a Proc (lambda {...}), and
   #   retry (:retry) are ignored because flow of the program is allowed to
   #   continue for each of those, whereas a :raise will abort processing of
   #   subsequent tasks until caught.
-  # * If a task raises an Exception, but the task is enclosed in another task
+  # * If a title raises an Exception, but the title is enclosed in another title
   #   that merely logs the Exception, then the @errors attribute is still
   #   incremented.  This is useful for signalling when a warning condition
   #   occured, but the program is able to continue the processing of
   #   subsequent tasks.
   attr_reader :errors
   
-  # Tracks the number of times an Exception has been raised, regardless of
-  # whether the Exception was re-thrown.
-  attr_reader :warnings
-
-  # Array of actions to perform when an Exception is thrown by code in the
-  # yield block.
-  # * The Watcher instance maintains a default @fail_actions Array for use
-  #   when a given invocation does not specify a different set of actions to
-  #   perform.
-  # * The default value is an Array of two actions, [:log, :raise], which
-  #   notes the error in the logs, and re-throws the Exception.
-  attr_accessor :fail_actions
-  
-  # A String to include inside the log entry when an Exception is thrown.
-  # * This may be required in some circumstances where a separate log
-  #   monitoring program is scanning log files looking for a string such as,
-  #   ERROR or FAILURE.
-  # * The default value is nil.
-  attr_accessor :fail_symbol
-
-  # The number of spaces to indent a child level task under its parent task.
-  # * A Boolean value of false or an Integer value of 0 turns off indentation.
-  # * Any other value will set how many characters will be used for indenting
-  #   the logs.
-  # * This may only be set at the time the Watcher object is created.
-  # * The default value is 3.
-  attr_reader   :indent
-
   # A String of characters to use when joining multiple error lines into a
   # single line for output to the log.
   # * If nil, the Exception message is reported verbatim.  If not nil, the
@@ -61,11 +57,7 @@ class Watcher
   # * The default value is ' (LF) '.
   attr_accessor :merge_newlines
 
-  # Determines whether Watcher will display the task completion log events.
-  # * The default value is false.
-  attr_accessor :show_completion
-  
-  # Holds a Proc object assigned the task of formatting the system time to a
+  # Holds a Proc object assigned the title of formatting the system time to a
   # String for display.
   # * You can easily change how the times are displayed by passing a Proc
   #   object or a block to this attribute.
@@ -78,28 +70,27 @@ class Watcher
   attr_accessor :time_formatter
 
   # Determines level of reporting that Watcher object sends to the log file.
-  # * Possible values are the Symbols :debug, :info, :warn, :error, and
+  # * Possible values are the Symbols :debug, :verbose, :warn, :error, and
   #   :fatal.
-  # * The default value is :info.
+  # * The default value is :warn.
   attr_accessor :verbosity
 
   # Provide ability for Watcher to prioritize levels of verbosity.
   # * It wasn't designed to be directly used by client code, but I suppose it
   #   could be modified by consumer code in this script if the situation
   #   warranted.
-  @@verbosity_levels = {:debug=>0, :info=>1, :warn=>2, :error=>3, :fatal=>4}
-
-  # Stores the Strings used by Watcher to use when writting events to the log
-  # files.
-  # * Provided as a means to customize the output, say for language
-  #   localization purposes.
-  # * <b>Warning</b>, if you substitute your own Hash for @verbosity_strings
-  #   that does not have keys for the five levels of verbosity used by
-  #   Watcher, you may cause Watcher to start throwing exceptions with it
-  #   attempts to coerce the NilClass into a String.
-  # * The default value of @verbosity_strings is a Hash that maps the
-  #   verbosity levels to a reasonable String value.
-  attr_accessor :verbosity_strings
+  @@verbosity_levels = {:debug => 0, :verbose => 1, :always => 2}
+
+  # A String to include inside the log entry when an Exception is thrown.
+  # * This may be required in some circumstances where a separate log
+  #   monitoring program is scanning log files looking for a string such as,
+  #   ERROR or FAILURE.
+  # * The default value is nil.
+  attr_accessor :warn_symbol
+
+  # Tracks the number of times an Exception has been raised, regardless of
+  # whether the Exception was re-thrown.
+  attr_reader :warnings
 
   # We override the default accessibility for the :new method because we want
   # to ensure there is only one Watcher object created.
@@ -110,10 +101,11 @@ class Watcher
   # * You cannot use the #new method directly as you can with regular classes.
   # * Repeated calls to #Watcher.create simply return the same Watcher object.
   # * When a Watcher object is created, the default class initialization
-  #   values are usually perfectly suitable for the task at hand.  This is
+  #   values are usually perfectly suitable for the title at hand.  This is
   #   more-so true when just starting on a new project, when the default is to
-  #   send logging information to STDERR, which is usually what you want when
-  #   developing a program using the interactive method, such as with irb.
+  #   send logging information to STDOUT, which is usually what you want when
+  #   developing a program using the interactive method, such as with irb,
+  #   or when running a process from the command shell.
   # * It is easy to change any of the class attributes either at object
   #   instantiation, or while the program is being executed. During
   #   instantiation simply create a Hash object with a key Symbol for every
@@ -121,28 +113,28 @@ class Watcher
   #   instance, to create a globabl variable, aptly named <tt>$watcher</tt>:
   #
   # <tt>
-  # $watcher = Watcher.create({:fail_actions=>:warn, :verbosity=>:info})
+  # $watcher = Watcher.create(:default_fail=>:warn, :verbosity=>:warn)
   # </tt>
   #
   # * Each Watcher object instance stores an IO object for buffering data to
   #   the output log.  This can either be a regular file, STDERR, STDOUT, or
-  #   perhaps a FIFO or a pipe.  If you do not want to use STDERR or a
+  #   perhaps a FIFO or a pipe.  If you do not want to use STDOUT or a
   #   regular file, simply make the connection, and pass it to your Watcher
   #   instance.
-  # * The default value of @output is an IO instance of STDERR.
+  # * The default value of @output is an IO instance of STDOUT.
   def Watcher.create(attributes={})
     @@watcher ||= new(attributes)
   end
 
-  # Used to identify a low-level task during development.
+  # Used to identify a low-level title during development.
   # * This is most frequently invoked without a block, in which case it simply
-  #   appends the task start and stop messages to the log if the verbosity
+  #   appends the title start and stop messages to the log if the verbosity
   #   level is at :debug.
   # * Use this when in lieu of a group of 'puts' statements littered through
   #   your source code.
   # * See detailed decription for #monitor for how this method works.
-  def debug(task, fail_actions=@fail_actions)
-    monitor(task,:debug,block_given?,fail_actions) { yield }
+  def debug(title, event_fail=@default_fail)
+    monitor(title, :debug, block_given?, event_fail) { yield }
   end
 
   # Usually the method of choice for your general logging purpose
@@ -152,24 +144,30 @@ class Watcher
   # * I recommend using this for every level of program logic that you desire
   #   to track.
   # * See detailed decription for #monitor for how this method works.
-  def info(task, fail_actions=@fail_actions)
-    monitor(task,:info,block_given?,fail_actions) { yield }
+  def verbose(title, event_fail=@default_fail)
+    monitor(title, :verbose, block_given?, event_fail) { yield }
+  end
+
+  # Use this sparingly.  (User specifies this verbosity level so your program
+  # will not show anything, unless there is an error.)
+  def always(title, event_fail=@default_fail)
+    monitor(title, :always, block_given?, event_fail) { yield }
   end
 
   # Each Watcher object instance can optionally send a copy of its
   # log file to a given email account.
   # * You might want to call this if there were any errors while completing
   #   your tasks.
+  # * NOTE: It is the responsibility of the calling routine to send the logs
+  #   to a file if you intend to send the email.
+  #   $watcher = Watcher.create(:output=>log_file, ...)
   def send_email(email,subject)
     @output.flush    # flush the output buffer
-    system "mail -s '#{subject}' #{email} < #{@output_path}"
-
-    # reopen the output buffer for append mode
-    @output = File.open(@output_path, 'a')
-    unless @output # Redirect to STDERR if open failed.
-      @output = STDERR
-      @debug_out << "could not open [#{output}]. " + e.message + "\n" \
-        if WATCHER_DEBUG
+    if File.file?(@output_path)
+      system "mail -s '#{subject}' #{email} < #{@output_path}"
+    else
+      msg = "Cannot send log file as email because no log file was created"
+      raise ArgumentError.new(msg)
     end
   end
 
@@ -187,8 +185,8 @@ class Watcher
     if WATCHER_DEBUG
       # Open our dump file, and let Ruby environment close it when we exit.
       begin
-        # mode 'w' truncates file if not empty.
-        @debug_out = File.open(WATCHER_DEBUG_OUTPUT_FILENAME,'w')
+        # mode 'a' appends to file, and 'w' truncates file if not empty.
+        @debug_out = File.open(WATCHER_DEBUG_OUTPUT_FILENAME,'a')
       rescue Exception => e
         # if we couldn't open it, just send debugging data to STDERR,
         # and insert a message indicating the failure.
@@ -196,14 +194,18 @@ class Watcher
         @debug_out << "could not open [#{WATCHER_DEBUG_OUTPUT_FILENAME}]. " \
            + e.message + "\n"
       end
+      @debug_out << "****************************************\n"
+      @debug_out << Time.now.to_s + "* starting " + File.basename($0) + "\n"
+      @debug_out.flush unless @debug_out == STDERR or @debug_out == STDOUT
     end
     
     # Watcher debugging facility
     if WATCHER_DEBUG and WATCHER_DEBUG_ATTRIBUTES
       @debug_out << "=== Explicitly requested attributes: \n"
       attributes.keys.each do |k|
-        @debug_out << "\t:#{k}=>#{attributes[k].inspect}\n"
+        @debug_out << "\t:#{k} => #{attributes[k].inspect}\n"
       end
+      @debug_out.flush unless @debug_out == STDERR or @debug_out == STDOUT
     end
 
     # Here we define the default values for a newly created Watcher object.
@@ -213,168 +215,122 @@ class Watcher
     # * Any keys provided by the user simply cause the associated key
     #   values provided by the consumer code to overwrite the values
     #   provided in the literal Hash below.  Isn't Ruby fun?
-    attributes.delete(:output) if attributes[:output] == nil
-    attributes={:fail_actions=>[:log,:raise], :fail_symbol=>nil, :indent=>3,
-      :merge_newlines=>' (LF) ', :output=>STDERR, :show_completion=>false,
-      :time_formatter=>lambda { Time.now.to_s }, :verbosity=>:info,
-      :verbosity_strings=>{:debug=>' DEBUG:',:info=>' INFO :',
-        :warn=>' WARN :',:error=>' ERROR:',
-        :fatal=>' FATAL:'}
+#    attributes.delete(:output) if attributes[:output] == nil
+    attributes.delete(:output_path) if attributes[:output_path] == nil
+    attributes={
+      :default_fail => {:failure => :error},
+      :error_symbol => nil, :warn_symbol => nil,
+      :merge_newlines => ' (LF) ', :output_path => STDOUT,
+      :time_formatter => lambda { Time.now.to_s },
+      :verbosity => :always,
+      :write => :overwrite
     }.merge(attributes)
     
     # Watcher debugging facility
     if WATCHER_DEBUG and WATCHER_DEBUG_ATTRIBUTES
       @debug_out << "=== Merged attributes: \n" << "\n"
       attributes.keys.each do |k|
-        @debug_out << "\t:#{k}=>#{attributes[k].inspect}\n"
+        @debug_out << "\t:#{k} => #{attributes[k].inspect}\n"
       end
+      @debug_out.flush unless @debug_out == STDERR or @debug_out == STDOUT
     end
     
+    # initialize public attributes
+    @errors = 0
+    @warnings = 0
+
+    # initialize private attributes
+    @last_hier = nil               # no last events at start
+    @event_relationship = :child    # first event will be a child of...nil
+    
+    # Array of Event instances for working with nested levels.
+    @events = Array.new
+    
     # Here the product of the merges Hashes is used to populate the
     # instantiated Watcher object attributes.
     # Nothing to see here, move on...  Unless you want to improve
     # Watcher...
-    @current_indent=0
-    @errors=0
-    @warnings=0
-    @fail_actions=attributes[:fail_actions]
-    @fail_symbol=attributes[:fail_symbol]
-    @indent=attributes[:indent]
-    @merge_newlines=attributes[:merge_newlines]
-    @output=attributes[:output]
-    @show_completion=attributes[:show_completion]
-    @time_formatter=attributes[:time_formatter]
-    @verbosity=attributes[:verbosity]
-    @verbosity_strings=attributes[:verbosity_strings]
+    @default_fail = attributes[:default_fail]
+    @error_symbol = attributes[:error_symbol]
+    @warn_symbol = attributes[:warn_symbol]
+    @merge_newlines = attributes[:merge_newlines]
+    @output_path = attributes[:output_path]
+    @time_formatter = attributes[:time_formatter]
+    @verbosity = attributes[:verbosity]
+    @write_mode = attributes[:write]
+    
+    # after the attribute merge, perform sanity checks
+    if @time_formatter.class != Proc
+      raise ArgumentError(":time_formatter must be a Proc instance")
+    end
+    
+    # make sure verbosity makes sense
+    unless @@verbosity_levels.include? @verbosity
+      msg = ":verbosity must be one of ["
+      @@verbosity_levels.keys.each { |k| msg << k.inspect << ' ' }
+      msg << ']'
+      raise ArgumentError.new(msg)
+    end
+    
+    # verify file output mode
+    @write_mode = case @write_mode
+    when :append
+      'a'
+    when :overwrite
+      'w'
+    else
+      msg = "Watcher.initialize: :write must be one of either :overwrite" \
+        + " or :append, not #{@write_mode.inspect}."
+      raise ArgumentError.new(msg)
+    end
     
     # Watcher debugging facility
     if WATCHER_DEBUG and WATCHER_DEBUG_ATTRIBUTES
       @debug_out << "=== Instance attributes: \n" << "\n"
-      @debug_out << "    @current_indent\t\t=#{@current_indent}" << "\n"
-      @debug_out << "    @fail_actions\t\t=#{@fail_actions.inspect}" << "\n"
-      @debug_out << "    @fail_symbol\t\t=\"#{@fail_symbol}\"" << "\n"
-      @debug_out << "    @indent\t\t\t=#{@indent}" << "\n"
+      @debug_out << "    @default_fail\t\t=#{@default_fail.inspect}" << "\n"
+      @debug_out << "    @error_symbol\t\t=\"#{@error_symbol}\"" << "\n"
+      @debug_out << "    @warn_symbol\t\t=\"#{@warn_symbol}\"" << "\n"
       @debug_out << "    @merge_newlines\t\t=#{@merge_newlines}" << "\n"
-      @debug_out << "    @output\t\t\t=#{@output}" << "\n"
-      @debug_out << "    @show_completion\t\t=#{@show_completion}" << "\n"
+      @debug_out << "    @output_path\t\t\t=#{@output_path}" << "\n"
       @debug_out << "    @time_formatter\t\t=#{@time_formatter}" << "\n"
       @debug_out << "    @verbosity\t\t\t=:#{@verbosity}" << "\n"
-      @debug_out << "    @verbosity_strings\t\t=" \
-        + "#{@verbosity_strings.inspect}\n"
+      @debug_out.flush unless @debug_out == STDERR or @debug_out == STDOUT
     end
     
     # Open the log file if not STDERR nor STDOUR
-    if @output != STDERR and @output != STDOUT
-      @output_path = File.expand_path(@output)
-      # mode 'w' truncates file if not empty.
-      @output = File.open(@output_path, 'w')
-      unless @output # Redirect to STDERR if open failed.
-        @output = STDERR
-        @debug_out << "could not open [#{output}]. " + e.message + "\n" \
-          if WATCHER_DEBUG
+    if @output_path != STDERR and @output_path != STDOUT
+      @output_path = File.expand_path(@output_path)
+      @output = File.open(@output_path, @write_mode)
+      unless @output # Redirect to STDOUT if open failed.
+        @output = STDOUT
+        @debug_out << "could not open [#{output_path}]. " + e.message \
+          + "\n" if WATCHER_DEBUG
       end
+    else
+      @output = @output_path
     end
   end # initialize
   
-  # Returns a String representing the indentation characters to use
-  # for the event.
-  # * A new event increments the indentation level.
-  # * A done event and an error event decrements the indentation level.
-  # * A nil event represents a note, with no potential child-tasks. (No
-  #   indentation change is required.)
-  def get_log_indentation_string(event=nil)
-    if @indent.class != Fixnum # don't indent unless numberical value
-      case event
-      when nil then result = '='
-      when :new then result = '>'
-      when :done then result = '<'
-      when :error then result = '*'
-      else
-        result = '='
-        @debug_out << " =WD= Invalid event Symbol = #{event.inspect}\n" \
-          if WATCHER_DEBUG
-      end
-    else # we got a Fixnum, so use indented log output
-      result = ' ' # plan to place at least one space after the time mark
-      # If WATCHER_DEBUG, we want to use periods instead of spaces to indent
-      space = '.'
-      unless WATCHER_DEBUG
-        # create some run-time protection from bad input
-        # TODO: Validate data when attributes accessors invoked
-        @current_indent = 0 if @current_indent < 0
-        @indent = 3 if @indent < 3
-        space = ' ' # if not debugging Watcher, go ahead and use spaces
-      end
-      
-      #--
-      # If @indent is >= 3, arrows are created at each indent level to show
-      # the parent-child relationship between tasks.
-      # * head refers to the head of a log event arrow, and
-      # * tail refers to the tail of a log event arrow.
-      #++
-      case event
-      when nil
-        # same as :new but do not increment the level at the end
-        @current_indent.times do
-          result += space * (@indent-1)
-        end
-        result += 'I'  # tail
-        result += '=' * (@indent-2)
-        result += '>'  # head
-      when :new
-        # increate indent after make this String
-        @current_indent.times do
-          result += space * (@indent-1)
-        end
-        result += 'N'  # tail
-        result += '-' * (@indent-2)
-        result += '>'  # head
-        @current_indent += 1
-      when :done
-        # decrease indent before make this String
-        @current_indent -= 1
-        @current_indent.times do
-          result += space * (@indent-1)
-        end
-        result += 'C'   # head
-        result += '-' * (@indent-1)
-      when :error
-        # decrease indent before make this String
-        @current_indent -= 1
-        @current_indent.times do
-          result += space * (@indent-1)
-        end
-        result += 'E'   # head
-        result += '-' * (@indent-2)
-      else
-        # invalid event code invoked from within Watcher.monitor
-        @debug_out << " =WD= Invalid event Symbol = #{event.inspect}\n" \
-          if WATCHER_DEBUG
-      end
-    end
-    result
-  end
-  
   # Get the time String, but protect against if plug-in Proc causes an
   # Exception.
   def get_protected_log_time_string
-    begin
+    @time = begin
       @time_formatter.call
     rescue
-      Time.now.utc.to_s + " == WARNING: Time Formatter raised Exception! == "
+      Time.now.to_s + " (WARNING! :time_formatter Proc raised #{$!.class} exception) "
     end
   end
   
-  # When either #debug or #info is invoked, it ends up passing the
+  # When either #debug or #verbose is invoked, it ends up passing the
   # batton to the private #monitor method.
   # * #monitor first sets up some local variables, including setting up a
-  #   variable to store the effective fail_actions for reasons decribed below,
-  #   then optionally makes a log entry concerning the task you desire to
+  #   variable to store the effective default_fail for reasons decribed below,
+  #   then optionally makes a log entry concerning the title you desire to
   #   execute.  This log entry only takes place if the verbosity level of the
-  #   task is equal to or exceeds the Watcher object's stored verbosity
+  #   title is equal to or exceeds the Watcher object's stored verbosity
   #   minimum level, @verbosity.
   # * After preparations are complete, Watcher checks whether you sent the
-  #   #debug or #info method a block to execute. If you did not send a block,
+  #   #debug or #verbose method a block to execute. If you did not send a block,
   #   Watcher simply returns control to your code at the point immediately
   #   following where you invoked Watcher.
   # * If you did pass a block to execute, Watcher sets up a
@@ -382,215 +338,188 @@ class Watcher
   #   block inside that protective environment.
   # * If code invoked by that block triggers an Exception, Watcher passes
   #   control to the #protect method, which orchestrates a recovery path based
-  #   on the consumer provided fail_actions Array.  If the #debug or #info
+  #   on the consumer provided default_fail Array.  If the #debug or #verbose
   #   methods were not given a specific set of actions to perform if an
   #   Exception occurs, then the Watcher object uses its object instance
-  #   attribute @fail_actions to provide the list of actions to complete.  For
+  #   attribute @event_fail to provide the list of actions to complete.  For
   #   information how this works, see the documentation for the #protect
   #   method.
-  def monitor(task,run_level,block_was_given,fail_actions) #:doc:
-    # Watcher debugging facility
-    if WATCHER_DEBUG and WATCHER_DEBUG_FAILACTIONS
-      @debug_out << "fail_actions = #{fail_actions.inspect}\n"
+  def monitor(title, run_level, block_was_given, event_fail) #:doc:
+    # Why have @events, an Array of events?
+    # * When new title is started, if its title level meets or exceeds the
+    #   Watcher verbosity level, the @events Array is cleared and the event
+    #   is sent to the log.  The @events Array must be cleared because
+    #   preceeding indentation will not otherwise line up.
+    # * When new title is started, if its title level is lower than the Watcher
+    #   verbosity level, its event Hash is appended to the @events Array for
+    #   later log replay if exception is thrown.
+    # * When title is successfully completed, its matching event Hash is
+    #   located (from the right) of the @events Hash, and it and all following
+    #   events are removed from the @events Array.
+    # * When title is unsuccessfully completed, a event replay is performed and
+    #   all events in the @events Hash are sent to the logs regardless of
+    #   their event level.  This way an understandable stack-frame replay is
+    #   possible to determine the actual cause of the exception.
+
+    if WATCHER_DEBUG and WATCHER_DEBUG_FAIL_ACTIONS
+      @output.puts "WATCHER MONITOR -- FAIL_ACTIONS for [#{title}]"
+      event_fail.each { |k,v| @output << " #{k.inspect} == #{v.inspect}\n" }
     end
     
-    # assume no exception for this task
-    success_flag = true
+    # Validate input; abort if input parameters are not clean.
+    #   (@time, @last_hier, @title, @fail_actions, @relationship)
+    event = Event.new(get_protected_log_time_string, @last_hier, title, \
+      event_fail, @event_relationship)
     
-    # validate input upon initial invocation, not when handling Exception...
-    # abort task if input parameters are not clean
-    if not task.respond_to?(:to_s)
-      @output << get_protected_log_time_string
-      @output << "WATCHER: the 'task' must be printable!\n"
-      return # abort task
-    elsif not @@verbosity_levels.has_key?(run_level)
-      @output << get_protected_log_time_string + "WATCHER: The given " \
-        + "run_level, #{runlevel.inspect}, is not a member of " \
-        + "Watcher::@@verbosity_levels.  Task = [#{task}]\n"
-        return # abort task
-    else
-      # always validate the requested fail actions by helper method
-      protect(task, fail_actions) # e=nil, therefore just validate
+    # Validate input; abort if input parameters are not clean.
+    unless @@verbosity_levels.has_key? run_level
+      # Defensive programming
+      msg = "Watcher.monitor: The second argument (run_level) must be one" \
+        + ' of '  + @@verbosity_levels.inspect + ".  Your argument was" \
+        + run_level.inspect + ". Aborting [#{title}]"
+      raise ArgumentError.new(msg)
     end
     
-    # Watcher debugging facility
-    if WATCHER_DEBUG and WATCHER_DEBUG_LEVELS
-      @debug_out << get_protected_log_time_string \
-        + " =WD= task level = :#{run_level}, " \
-        + "verbosity level = :#{@verbosity}\n"
+    # Either append this event to @events Array, or send it to the log
+    if @@verbosity_levels[run_level] < @@verbosity_levels[@verbosity]
+      # This event is lower than our minimum verbosity, so store it in our
+      # Array for replay if needed later.
+      @events << event
+      @debug_out << "* HIDDEN: #{event}\n" if WATCHER_DEBUG and WATCHER_DEBUG_VERBOSITY
+    else
+      # forget previous replays since this gets printed
+      @debug_out << "* VISIBLE:#{event.to_s} (*) clearing previous #{@events.size} events from array\n" if WATCHER_DEBUG and WATCHER_DEBUG_VERBOSITY
+      @events = Array.new
+      
+      # NOTE:  event.to_s retuns time-stamp then its event hier
+      @output << event.to_s + "\n"
+      @output.flush # Do this always, not just 'if WATCHER_DEBUG'
     end
+
+    # If event has a block, protect and execute the block
+    if block_was_given # block_given? from #debug or #verbose
+      if WATCHER_DEBUG and WATCHER_DEBUG_OUTPUT and false
+        @output << " +monitor_block(#{event.title})"
+      end
+      monitor_block(event) { yield }
+    end # block given
     
-    # log the event if the task run level >= verbosity
-    if @@verbosity_levels[run_level] >= @@verbosity_levels[@verbosity]
-      output = get_protected_log_time_string
-      # if no new block given, then task event = nil, (don't indent)
-      output << get_log_indentation_string(block_was_given ? :new : nil)
-      output << task << "\n"
-      @debug_out << output if WATCHER_DEBUG and WATCHER_DEBUG_OUTPUT
-      @output << output
+    # Once this Event is complete, make the next event a sibling of this event
+    @event_relationship = :sibling
+    @last_hier = event.hier.dup
+
+    if WATCHER_DEBUG and WATCHER_DEBUG_VERBOSITY
+      @debug_out << "(L) @last_hier = " + @last_hier + "\n"
+      @debug_out.flush
     end
+  end
+  
+  def monitor_block (event)
+    # During execution of the parameterized block, the following events must
+    # be children events of this event.  After the block completes, following
+    # events are sibling events of this event.
+    @event_relationship = :child
+    @last_hier = event.hier.dup
+
+    # Event knows its own max tries value
+    tries_remaining = event.tries
     
-    # okay, preparations complete, if a block was given, do it!
-    if block_was_given # block_given? from #debug or #info
-      begin
-        # Create a Continuation object.  A :retry fail action will come back
-        # to this point in code.  A classic "begin, rescue, retry" does not
-        # work, because the retry would occur from an invoked method, protect.
-        callcc{|@continuation|}
-        yield
-        # Log the event if the task run level is greater or equal to the
-        # verbosity
-        # * The task complete log entry only takes place after your block
-        #   returns.
-        # * If you did not send a task block to complete, Watcher will
-        #   conditionally log the event based on the verbosity level, then
-        #   exit back to your code just after you invoked Watcher.
-        # * PROGRAMMER NOTE:  I moved the code from the ensure block to just
-        #   after the yield code in the begin block.
-        # * The reason for this is that there are really two logical ways
-        #   to complete a task: (1) successfully, and (2) unsuccessfully.
-        #   1. This code represents the path by successful code, while the
-        #      rescue block represents the patch by unsuccessful code.
-        #   2. Because the ensure block is always taking place, a failed block
-        #      of consumer code would log the error, then enter the ensure
-        #      block and log the task complete.  That gives the wrong
-        #      impression.
-        if @@verbosity_levels[run_level] >= @@verbosity_levels[@verbosity]
-          # Only display if user wants logging of compeltion events
-          # * Although we don't show the line when @show_completion is false,
-          #   we must still call #get_log_indentation_string(:done) to adjust
-          #   the @current_indent attribute properly.
-          output << get_protected_log_time_string
-          output << get_log_indentation_string(:done) << task << "\n"
-          if @show_completion
-            @debug_out << output if WATCHER_DEBUG and WATCHER_DEBUG_OUTPUT
-            @output << output
-          end
-        end
-      rescue
-        # catch any Exceptions thrown by the executed block
-        success_flag = protect(task, fail_actions, $!)
+    begin
+      tries_remaining -= 1
+      yield
+    rescue # from exception thrown during original yield
+      # Start by sending all queued (hidden) events to the log. (They were
+      # only queued up so we could log them in event of a failure.)
+      # NOTE: @events could be empty...that's okay.
+      @events.each do |e|
+        @output << e.to_s
+        @output << " (flushed)" if WATCHER_DEBUG and WATCHER_DEBUG_OUTPUT
+        @output << "\n"
       end
-    end
-    
-    # return whether no exceptions occured
-    success_flag
-  end
+      # After we output all hidden events, we erase the @events Array,
+      # so we don't output them on the next retry failure.
+      @events = Array.new
+      
+      @output << get_protected_log_time_string + ': ' \
+        + event.hier + ' ***'
 
-  # Iterate through fail_actions parameter for a task, and for each action,
-  # either perform or simply ensure it's a valid action.
-  # * #protect is called by #monitor when your code invokes #debug or #info in
-  #   order to validate the fail_actions parameter.
-  # * #protect is also called by #monitor when the associated code block
-  #   triggers an Exception. It receives an Array of actions to perform,
-  #   <tt>fail_actions</tt>, and executes each one in turn.
-  # * Perhaps the most complex--however reasonable--example would be if you
-  #   wanted to log the event, then invoke some snippet of code to attempt to
-  #   resolve the problem, then retry the block.  This can be done by
-  #   specifying the following fail actions. <b>(Be careful, however, as this
-  #   can easily lead to an infinite loop if the recovery Proc does not
-  #   resolve the cause of the original Exception!)</b>
-  # * This is meant to provide you a bit of flexibility on how to handle
-  #   conditions, however a complex Array of fail_actions may defeat your goal
-  #   of code simplification.
-  def protect(task, fail_actions, e=nil) #:doc:
-    # * If e is nil, then return true if valid group, false or raise
-    #   exception if invalid fail action found.
-    # * If e is an Exception, perform required actions.
-
-    # only increase warning if also performing actions
-    if e != nil
-      if e.kind_of?(Exception)
-        @warnings += 1
+      # Place error or warning flag in the log
+      if tries_remaining == 0 and event.failure == :error
+        if @error_symbol != nil and @error_symbol != ''
+          @output << ' ' + @error_symbol
+        end
       else
-        err_msg = "invalid exception instance (Watcher.monitor error)"
-        raise ArgumentError.new(err_msg)
+        if @warn_symbol != nil and @warn_symbol != ''
+          @output << ' ' + @warn_symbol
+        end
       end
-      @debug_out << "e = #{e.inspect}\n" if WATCHER_DEBUG
-    end
-    
-    if fail_actions.class == Symbol or fail_actions.class == Proc
-      # NOTE: we pass the fail_actions (plural) parameter
-      protect_action(task, fail_actions, e)
-    elsif fail_actions.class == Array
-      fail_actions.each do |fail_action|
-        # NOTE: we pass the fail_action (singular) loop iteration variable
-        protect_action(task, fail_action, e)
+
+      # Append to log result of merging newlines from exception message
+      if @merge_newlines != nil and @merge_newlines != ''
+        @debug_out.puts "[#{$!.message}]" if WATCHER_DEBUG and WATCHER_DEBUG_OUTPUT
+        message = $!.message.split(/\n/).join @merge_newlines
       end
-    else
-      err_msg = "unsupported fail action for task = [#{task}] " \
-        + "(must be Symbol or Array of Symbols and Procs)"
-      raise ArgumentError.new(err_msg)
-    end
-  end
+      @output << ' ' + message + "\n"
+      @output.flush
 
-  # Either perform action, or simply validate it's valid
-  def protect_action(task, fail_action, e)
-    # * If e is nil, then validate the action against the possible
-    #   actions.  Raise ArgumentError Exception if invalid action found.
-    # * If e is a kind of Exception, perform required action.
-
-    # only time-stamp if also performing actions
-    if e != nil
-      # create the time stamp and indentation portion of the log entry
-      log_base = get_protected_log_time_string \
-        + get_log_indentation_string(:error)
-      @debug_out << "e = #{e.inspect}\n" if WATCHER_DEBUG
-    end
+      if tries_remaining > 0
+        begin
+          # Call the user-supplied recovery Proc with the original exception
+          event.proc.call($!)
+        rescue # from exception during user's recovery proc
+          # If user-supplied Proc caused an Exception, then bail
+          @errors += 1
+          raise $!
+        end
 
-    if fail_action.class == Proc
-      if e != nil
-        # return result from user-specified code
-        return fail_action.call(e) # send the Exception instance
-      end
-    elsif fail_action.class == Symbol
-      case fail_action
-      when :log
-        if e != nil
-          output = log_base + '*' + task + '*' + ' ==> '
-          output << "#{@fail_symbol} ==> " if @fail_symbol
-          if @merge_newlines != false
-            output << e.message.split(/\n/).join(@merge_newlines)
-          else # no merge, but multiple lines
-            output << e.message
-          end
-          @output << output << "\n"
+        @output << get_protected_log_time_string + ': ' \
+          + event.hier + ' ### '
+        @output << case tries_remaining
+        when 1
+          "(1 try left)\n"
+        else
+          '(' + tries_remaining.to_s + " tries left)\n"
+        end
+        @output.flush
+
+        retry
+      else # no more retries remaining
+        if not $!.kind_of? Exception
+          raise "$! no longer an exception [#{$!.inspect}]"
         end
-      when :raise
-        if e != nil
+        if event.failure == :error
           @errors += 1
-          raise
+          raise $!
+        else
+          @warnings += 1
+          @output.flush
         end
-      when :retry # FIXME: this is not well tested
-        @continuation.call if e != nil
-      else
-        err_msg = "unsupported fail action Symbol for task = [#{task}] " \
-          + "(must be one of [:log, :raise, :retry])"
-        raise ArgumentError.new(err_msg)
-      end
-    else # only Symbols and Procs can be designated fail actions
-      raise ArgumentError.new("invalid fail action (Watcher.protect error)")
-    end
+      end # perform a retry attempt
+    ensure
+      # Once this Event is complete, make the next event a sibling of this event
+      @event_relationship = :sibling
+      @last_hier = event.hier.dup
+    end # rescue from top tier exception
   end
-  
-  # Set to true when Watcher should execute itself in debugging mode.
-  WATCHER_DEBUG=false
 
-  # Name of file that Watcher creates for its debugging output.
-  WATCHER_DEBUG_OUTPUT_FILENAME='watcher-debug.log'
+  # Watcher version
+  VERSION = '1.2.0'
 
-  # Set to true when debugging Watcher instance attribute settings.
-  WATCHER_DEBUG_ATTRIBUTES=false
-  
-  # Set to true when debugging Watcher handling of fail actions.
-  WATCHER_DEBUG_FAILACTIONS=false
+  # Set to true when Watcher should execute itself in debugging mode.
+  WATCHER_DEBUG = false
 
+  # Set to true when debugging Watcher failure actions.
+  WATCHER_DEBUG_FAIL_ACTIONS = false
+  
   # Set to true when debugging Watcher verbosity levels.
-  WATCHER_DEBUG_LEVELS=false
+  WATCHER_DEBUG_VERBOSITY = false
 
-  # Set to true when debugging Watcher.monitor.
-  WATCHER_DEBUG_MONITOR=false
+  # Name of file that Watcher creates for its debugging output.
+  WATCHER_DEBUG_OUTPUT_FILENAME = 'watcher-debug.log'
 
+  # Set to true when debugging Watcher instance attribute settings.
+  WATCHER_DEBUG_ATTRIBUTES = false
+  
   # Set to true when debugging Watcher's log strings.
-  WATCHER_DEBUG_OUTPUT=false
+  WATCHER_DEBUG_OUTPUT = false
 end