sawmill 0.0.1

data/History.rdoc

@@ -0,0 +1,3 @@
+=== 0.0.1 / 2009-10-28
+
+* Initial test release

data/README.rdoc

@@ -0,0 +1,81 @@
+== Sawmill
+
+Sawmill is a logging and log analysis system for Ruby.
+It extends the basic Ruby logging facility with log records and logfile parsing features.
+
+=== What is it?
+
+A user reported a bug in your Rails application, and now you have to figure out what happened. You can spend hours poring over megabytes of text log files, stretching your grep-fu to its snapping point... or you can use Sawmill.
+
+Your CEO wants to know how many people viewed the new product pages last week. You can write yet another arcane sed script looking for that elusive pattern in the log files... or you can use Sawmill.
+
+Sawmill is an extension to the standard ruby Logger mechanism that enables automated log analysis. Designed especially for web applications using Ruby on Rails and similar frameworks, Sawmill originated in the source to www.geopage.com, where it has been analyzing the production Rails logs for more than three years.
+
+=== Feature list
+
+* Drop-in compatible with standard ruby Logger class
+* Custom log levels
+* Process log entries as objects
+* Group log entries into log records with unique record IDs and attributes
+* Filtering and analysis of log entries and log records
+* Write logfiles, optionally with one of several logfile rotation mechanisms
+* Parse logfiles back into log entries and log records
+* Log analysis framework
+* Rack middleware for quick setup of web service logs
+
+=== Requirements
+
+* Ruby 1.8.6 or later (1.8.7 recommended), Ruby 1.9.1 or later, or JRuby 1.4 or later.
+* versionomy gem.
+
+=== Installation
+
+ gem install sawmill
+
+=== Known issues and limitations
+
+Sawmill is currently under development, and some features are not yet complete or fully tested.
+
+=== Development and support
+
+Documentation is available at http://virtuoso.rubyforge.org/sawmill/README_rdoc.html
+
+Source code is hosted on Github at http://github.com/dazuma/sawmill
+
+Report bugs on Github issues at http://github.org/dazuma/sawmill/issues
+
+Contact the author at dazuma at gmail dot com.
+
+=== Author / Credits
+
+Lumber is written by Daniel Azuma (http://www.daniel-azuma.com/).
+
+== LICENSE:
+
+Copyright 2009 Daniel Azuma.
+
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice,
+  this list of conditions and the following disclaimer.
+* Redistributions in binary form must reproduce the above copyright notice,
+  this list of conditions and the following disclaimer in the documentation
+  and/or other materials provided with the distribution.
+* Neither the name of the copyright holder, nor the names of any other
+  contributors to this software, may be used to endorse or promote products
+  derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.

data/Rakefile

@@ -0,0 +1,147 @@
+# -----------------------------------------------------------------------------
+# 
+# Sawmill Rakefile
+# 
+# -----------------------------------------------------------------------------
+# Copyright 2009 Daniel Azuma
+# 
+# All rights reserved.
+# 
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+# 
+# * Redistributions of source code must retain the above copyright notice,
+#   this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright notice,
+#   this list of conditions and the following disclaimer in the documentation
+#   and/or other materials provided with the distribution.
+# * Neither the name of the copyright holder, nor the names of any other
+#   contributors to this software, may be used to endorse or promote products
+#   derived from this software without specific prior written permission.
+# 
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+# POSSIBILITY OF SUCH DAMAGE.
+# -----------------------------------------------------------------------------
+
+
+require 'rake'
+require 'rake/clean'
+require 'rake/gempackagetask'
+require 'rake/testtask'
+require 'rake/rdoctask'
+require 'rdoc/generator/darkfish'
+
+require ::File.expand_path("#{::File.dirname(__FILE__)}/lib/sawmill.rb")
+
+
+# Configuration
+extra_rdoc_files_ = ['README.rdoc', 'History.rdoc']
+
+
+# Default task
+task :default => [:clean, :rdoc, :package, :test]
+
+
+# Clean task
+CLEAN.include(['doc', 'pkg'])
+
+
+# Test task
+::Rake::TestTask.new('test') do |task_|
+  task_.pattern = 'tests/tc_*.rb'
+end
+
+
+# RDoc task
+::Rake::RDocTask.new do |task_|
+  task_.main = 'README.rdoc'
+  task_.rdoc_files.include(*extra_rdoc_files_)
+  task_.rdoc_files.include('lib/sawmill/**/*.rb')
+  task_.rdoc_dir = 'doc'
+  task_.title = "Sawmill #{::Sawmill::VERSION_STRING} documentation"
+  task_.options << '-f' << 'darkfish'
+end
+
+
+# Gem task
+gemspec_ = ::Gem::Specification.new do |s_|
+  s_.name = 'sawmill'
+  s_.summary = 'Sawmill is a logging and log analysis system for Ruby.'
+  s_.version = ::Sawmill::VERSION_STRING
+  s_.author = 'Daniel Azuma'
+  s_.email = 'dazuma@gmail.com'
+  s_.description = 'Sawmill is a logging and log analysis system for Ruby. It extends the basic Ruby logging facility with log records and parsing abilities.'
+  s_.homepage = 'http://github.com/dazuma/sawmill'
+  s_.rubyforge_project = 'virtuoso'
+  s_.required_ruby_version = '>= 1.8.6'
+  s_.files = ::FileList['lib/**/*.rb', 'tests/**/*.rb', '*.rdoc', 'Rakefile'].to_a
+  s_.extra_rdoc_files = extra_rdoc_files_
+  s_.has_rdoc = true
+  s_.test_files = ::FileList['tests/tc_*.rb']
+  s_.platform = ::Gem::Platform::RUBY
+  s_.add_dependency('blockenspiel', '>= 0.2.1')
+  s_.add_dependency('versionomy', '>= 0.1.1')
+end
+::Rake::GemPackageTask.new(gemspec_) do |task_|
+  task_.need_zip = false
+  task_.need_tar = true
+end
+
+
+# Publish RDocs
+desc 'Publishes RDocs to RubyForge'
+task :publish_rdoc_to_rubyforge => [:rerdoc] do
+  config_ = ::YAML.load(::File.read(::File.expand_path("~/.rubyforge/user-config.yml")))
+  username_ = config_['username']
+  sh "rsync -av --delete doc/ #{username_}@rubyforge.org:/var/www/gforge-projects/virtuoso/sawmill"
+end
+
+
+# Publish gem
+task :publish_gem_to_rubyforge => [:package] do |t_|
+  v_ = ::ENV["VERSION"]
+  abort "Must supply VERSION=x.y.z" unless v_
+  if v_ != ::Sawmill::VERSION_STRING
+    abort "Versions don't match: #{v_} vs #{::Sawmill::VERSION_STRING}"
+  end
+  gem_pkg_ = "pkg/sawmill-#{v_}.gem"
+  tgz_pkg_ = "pkg/sawmill-#{v_}.tgz"
+  release_notes_ = ::File.read("README.rdoc").split(/^(==.*)/)[2].strip
+  release_changes_ = ::File.read("History.rdoc").split(/^(===.*)/)[1..2].join.strip
+  
+  require 'rubyforge'
+  rf_ = ::RubyForge.new.configure
+  puts "Logging in to RubyForge"
+  rf_.login
+  config_ = rf_.userconfig
+  config_["release_notes"] = release_notes_
+  config_["release_changes"] = release_changes_
+  config_["preformatted"] = true
+  puts "Releasing sawmill #{v_} to RubyForge"
+  rf_.add_release('virtuoso', 'sawmill', v_, gem_pkg_, tgz_pkg_)
+end
+
+
+# Publish gem
+task :publish_gem_to_gemcutter => [:package] do |t_|
+  v_ = ::ENV["VERSION"]
+  abort "Must supply VERSION=x.y.z" unless v_
+  if v_ != ::Sawmill::VERSION_STRING
+    abort "Versions don't match: #{v_} vs #{::Sawmill::VERSION_STRING}"
+  end
+  puts "Releasing sawmill #{v_} to GemCutter"
+  `cd pkg && gem push sawmill-#{v_}.gem`
+end
+
+
+# Publish everything
+task :publish => [:publish_gem_to_gemcutter, :publish_gem_to_rubyforge, :publish_rdoc_to_rubyforge]

data/lib/sawmill/entry.rb

@@ -0,0 +1,307 @@
+# -----------------------------------------------------------------------------
+# 
+# Sawmill log entry classes
+# 
+# -----------------------------------------------------------------------------
+# Copyright 2009 Daniel Azuma
+# 
+# All rights reserved.
+# 
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+# 
+# * Redistributions of source code must retain the above copyright notice,
+#   this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright notice,
+#   this list of conditions and the following disclaimer in the documentation
+#   and/or other materials provided with the distribution.
+# * Neither the name of the copyright holder, nor the names of any other
+#   contributors to this software, may be used to endorse or promote products
+#   derived from this software without specific prior written permission.
+# 
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+# POSSIBILITY OF SUCH DAMAGE.
+# -----------------------------------------------------------------------------
+;
+
+
+module Sawmill
+  
+  
+  # This module is a namespace for log entry classes.
+  
+  module Entry
+    
+    
+    # A log entry that doesn't conform to sawmill's format.
+    
+    class UnknownData
+      
+      
+      def initialize(line_)
+        @line = line_.to_s
+      end
+      
+      
+      # Returns <tt>:unknown_data</tt>
+      def type; :unknown_data end
+      
+      # The line in the logfile as a raw string
+      attr_reader :line
+      
+      
+      def to_s  # :nodoc:
+        "#{type}: #{@line}"
+      end
+      
+      def inspect  # :nodoc:
+        "#<#{self.class}:0x#{object_id.to_s(16)} line=#{@line.inspect}>"
+      end
+      
+      def eql?(obj_)  # :nodoc:
+        obj_.kind_of?(Entry::UnknownData) && obj_.line == @line
+      end
+      
+      def ==(obj_)  # :nodoc:
+        eql?(obj_)
+      end
+      
+      def hash  # :nodoc:
+        type.hash ^ @line.hash
+      end
+      
+    end
+    
+    
+    # A log entry containing a standard log message.
+    
+    class Message
+      
+      
+      def initialize(level_, timestamp_, progname_, record_id_, message_)
+        @level = level_
+        @timestamp = timestamp_
+        @progname = progname_.to_s
+        @progname.gsub!(/\s+/, '')
+        @record_id = record_id_.nil? ? nil : record_id_.to_s
+        @record_id.gsub!(/\s+/, '') if @record_id
+        @message = message_.to_s
+      end
+      
+      
+      # Returns <tt>:message</tt>
+      def type; :message; end
+      
+      # The log level as a Sawmill::Level object
+      attr_reader :level
+      # The timestamp as a Time object
+      attr_reader :timestamp
+      # The progname as a string
+      attr_reader :progname
+      # The record ID as a string
+      attr_reader :record_id
+      # The message as a string
+      attr_reader :message
+      
+      
+      def to_s  # :nodoc:
+        "#{type}: #{@message}"
+      end
+      
+      def inspect  # :nodoc:
+        "#<#{self.class}:0x#{object_id.to_s(16)} level=#{@level.name} time=#{@timestamp.to_s.inspect} progname=#{@progname.inspect} record_id=#{@record_id.inspect} message=#{@message.inspect}>"
+      end
+      
+      def eql?(obj_)  # :nodoc:
+        obj_.kind_of?(Entry::Message) && obj_.level == @level && obj_.timestamp == @timestamp && obj_.progname == @progname && obj_.record_id == @record_id && obj_.message == @message
+      end
+      
+      def ==(obj_)  # :nodoc:
+        eql?(obj_)
+      end
+      
+      def hash  # :nodoc:
+        type.hash ^ @level.hash ^ @timestamp.hash ^ @progname.hash ^ @record_id.hash ^ @message.hash
+      end
+      
+      
+    end
+    
+    
+    # A log entry signalling the beginning of a log record.
+    
+    class BeginRecord
+      
+      
+      def initialize(level_, timestamp_, progname_, record_id_)
+        @level = level_
+        @timestamp = timestamp_
+        @progname = progname_.to_s
+        @progname.gsub!(/\s+/, '')
+        @record_id = record_id_.nil? ? nil : record_id_.to_s
+        @record_id.gsub!(/\s+/, '') if @record_id
+      end
+      
+      
+      # Returns <tt>:begin_record</tt>
+      def type; :begin_record; end
+      
+      # The log level as a Sawmill::Level object
+      attr_reader :level
+      # The timestamp as a Time object
+      attr_reader :timestamp
+      # The progname as a string
+      attr_reader :progname
+      # The record ID as a string
+      attr_reader :record_id
+      
+      
+      def to_s  # :nodoc:
+        "#{type}: #{@record_id}"
+      end
+      
+      def inspect  # :nodoc:
+        "#<#{self.class}:0x#{object_id.to_s(16)} level=#{@level.name} time=#{@timestamp.to_s.inspect} progname=#{@progname.inspect} record_id=#{@record_id.inspect}>"
+      end
+      
+      def eql?(obj_)  # :nodoc:
+        obj_.kind_of?(Entry::BeginRecord) && obj_.level == @level && obj_.timestamp == @timestamp && obj_.progname == @progname && obj_.record_id == @record_id
+      end
+      
+      def ==(obj_)  # :nodoc:
+        eql?(obj_)
+      end
+      
+      def hash  # :nodoc:
+        type.hash ^ @level.hash ^ @timestamp.hash ^ @progname.hash ^ @record_id.hash
+      end
+      
+      
+    end
+    
+    
+    # A log entry signalling the end of a log record.
+    
+    class EndRecord
+      
+      
+      def initialize(level_, timestamp_, progname_, record_id_)
+        @level = level_
+        @timestamp = timestamp_
+        @progname = progname_.to_s
+        @record_id = record_id_.nil? ? nil : record_id_.to_s
+      end
+      
+      
+      # Returns <tt>:end_record</tt>
+      def type; :end_record; end
+      
+      # The log level as a Sawmill::Level object
+      attr_reader :level
+      # The timestamp as a Time object
+      attr_reader :timestamp
+      # The progname as a string
+      attr_reader :progname
+      # The record ID as a string
+      attr_reader :record_id
+      
+      
+      def to_s  # :nodoc:
+        "#{type}: #{@record_id}"
+      end
+      
+      def inspect  # :nodoc:
+        "#<#{self.class}:0x#{object_id.to_s(16)} level=#{@level.name} time=#{@timestamp.to_s.inspect} progname=#{@progname.inspect} record_id=#{@record_id.inspect}>"
+      end
+      
+      def eql?(obj_)  # :nodoc:
+        obj_.kind_of?(Entry::EndRecord) && obj_.level == @level && obj_.timestamp == @timestamp && obj_.progname == @progname && obj_.record_id == @record_id
+      end
+      
+      def ==(obj_)  # :nodoc:
+        eql?(obj_)
+      end
+      
+      def hash  # :nodoc:
+        type.hash ^ @level.hash ^ @timestamp.hash ^ @progname.hash ^ @record_id.hash
+      end
+      
+      
+    end
+    
+    
+    # A log entry containing a log record attribute.
+    
+    class Attribute
+      
+      
+      def initialize(level_, timestamp_, progname_, record_id_, key_, value_, operation_=nil)
+        @level = level_
+        @timestamp = timestamp_
+        @progname = progname_.to_s
+        @progname.gsub!(/\s+/, '')
+        @record_id = record_id_.nil? ? nil : record_id_.to_s
+        @record_id.gsub!(/\s+/, '') if @record_id
+        @key = key_.to_s
+        @key.gsub!(/\s+/, '')
+        @value = value_.to_s
+        @operation = operation_ || :set
+      end
+      
+      
+      # Returns <tt>:attribute</tt>
+      def type; :attribute; end
+      
+      # The log level as a Sawmill::Level object
+      attr_reader :level
+      # The timestamp as a Time object
+      attr_reader :timestamp
+      # The progname as a string
+      attr_reader :progname
+      # The record ID as a string
+      attr_reader :record_id
+      # The operation, which can be :set, :append, :remove, :unset
+      attr_reader :operation
+      # The attribute key as a string
+      attr_reader :key
+      # The attribute value as a string
+      attr_reader :value
+      
+      
+      def to_s  # :nodoc:
+        "#{type}: #{@key}=#{@value.inspect}"
+      end
+      
+      def inspect  # :nodoc:
+        "#<#{self.class}:0x#{object_id.to_s(16)} level=#{@level.name} time=#{@timestamp.to_s.inspect} progname=#{@progname.inspect} record_id=#{@record_id.inspect} operation=#{@operation} key=#{@key.inspect} value=#{@value.inspect}>"
+      end
+      
+      def eql?(obj_)  # :nodoc:
+        obj_.kind_of?(Entry::Attribute) && obj_.level == @level && obj_.timestamp == @timestamp && obj_.progname == @progname && obj_.record_id == @record_id && obj_.key == @key && obj_.value == @value && obj_.operation == @operation
+      end
+      
+      def ==(obj_)  # :nodoc:
+        eql?(obj_)
+      end
+      
+      def hash  # :nodoc:
+        type.hash ^ @level.hash ^ @timestamp.hash ^ @progname.hash ^ @record_id.hash ^ @key.hash ^ @value.hash ^ @operation.hash
+      end
+      
+      
+    end
+    
+    
+  end
+  
+  
+end

data/lib/sawmill/entry_classifier.rb

@@ -0,0 +1,75 @@
+# -----------------------------------------------------------------------------
+# 
+# Sawmill entry sorter utility
+# 
+# -----------------------------------------------------------------------------
+# Copyright 2009 Daniel Azuma
+# 
+# All rights reserved.
+# 
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+# 
+# * Redistributions of source code must retain the above copyright notice,
+#   this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright notice,
+#   this list of conditions and the following disclaimer in the documentation
+#   and/or other materials provided with the distribution.
+# * Neither the name of the copyright holder, nor the names of any other
+#   contributors to this software, may be used to endorse or promote products
+#   derived from this software without specific prior written permission.
+# 
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+# POSSIBILITY OF SUCH DAMAGE.
+# -----------------------------------------------------------------------------
+;
+
+
+module Sawmill
+  
+  
+  # An object that classifies log entry objects, calling the proper methods
+  # on a Sawmill::EntryProcessor.
+  
+  class EntryClassifier
+    
+    
+    # Create a classifier that sends entries to the given
+    # Sawmill::EntryProcessor.
+    
+    def initialize(processor_)
+      @processor = processor_
+    end
+    
+    
+    # Call this method to classify a log entry and send it to the processor.
+    
+    def entry(entry_)
+      case entry_.type
+      when :unknown_data
+        @processor.unknown_data(entry_)
+      when :begin_record
+        @processor.begin_record(entry_)
+      when :end_record
+        @processor.end_record(entry_)
+      when :message
+        @processor.message(entry_)
+      when :attribute
+        @processor.attribute(entry_)
+      end
+    end
+    
+    
+  end
+  
+  
+end