beaver 0.0.1 → 1.0.0

data/README.rdoc

@@ -7,21 +7,14 @@ Beaver is a light DSL for parsing Rails production logs back into usable data. I
 * How many Widgets were created yesterday, and with what data?
 * Did anyone submit a form with the words "kill them all"? Yikes.
 
-Beaver is *not* intended as a replacement for something like Google Analytics, as it is trying to answer some differents kinds of questions.
-Rails logs contain some great information, but I've never found a good tool for using them programatically (except for speed analyzers).
-Hopefully Beaver can fill that niche for you. Personally I find it quite useful with Logwatch.
-
-Read the full documentation at http://jordanhollinger.com/docs/beaver/. The Beaver::Dam and Beaver::Request classes
-are probably what you're after.
+Read the full documentation at {jordanhollinger.com/docs/beaver/}[http://jordanhollinger.com/docs/beaver/].
+For a full list of matchers available to "hit", see the Beaver::Dam class. For a full list of methods available inside a "hit" block, or to members
+of the "hits" array in a "dam" block, see the Beaver::Request class.
 
 == Installation
-Beaver is still somewhat experimental, but it has proved stable and robust enough for a non-beta release.
-
   [sudo] gem install beaver
 
-== Use the command-line Beaver
-Write your beaver file:
-
+== Run a DSL file with beaver
   hit :failed_login, :method => :post, :path => '/login', :status => 401
 
   hit :new_widgets, :path => '/widgets', :method => :post, :status => 302 do
@@ -56,23 +49,7 @@ Run ''beaver'' from the command line, passing in your beaver file and some logs:
 
   beaver my_beaver_file.rb /var/www/rails-app/log/production.log*
 
-== Example use with Logwatch
-This assumes 1) you're rotating your Rails logs daily and 2) you're running logwatch daily.
-
-Check your beaver DSL file into your Rails app, maybe under /script.
-
-Add a logwatch config file for your new service at /etc/logwatch/conf/services/your_app.conf:
-
-  Title = "Your Rails App"
-  LogFile = NONE
-
-In /etc/logwatch/scripts/services/your_app:
-
-  beaver /var/www/your_app/script/beaver.rb --yesterday /var/www/your_app/log/production.log{,.1}
-
 == Use your own Beaver
-Use Beaver like a library:
-
   require 'rubygems'
   require 'beaver'
 
@@ -82,23 +59,43 @@ Use Beaver like a library:
 
   # Parse the logs, but only include requests from yesterday
   Beaver.parse logs, :on => Date.today-1 do
-    # Same DSL as above
     hit :failed_login, :method => :post, :path => '/login', :status => 401
     ...
   end
 
+== Use beaver like grep for Rails logs
+
+It's pretty difficult to grep through a multi-line log format and output each matching multi-line event. But Beaver is up to it:
+
+  beaver --path="/widgets.*" --method=post,put /var/www/rails-app/log/production.log
+
+Or format the output to a single line:
+
+  beaver --path="/widgets.*" --method=post,put /var/www/rails-app/log/production.log --print "%{ip} hit %{path} using %{method}"
+
+The command-line matchers are nearly identical to the DSL's. See more details with 'beaver --help'.
+
+== Example use with Logwatch
+This assumes 1) you're rotating your Rails logs daily and 2) you're running logwatch daily.
+
+Check your beaver DSL file into your Rails app, maybe under /script.
+
+Add a logwatch config file for your new service at /etc/logwatch/conf/services/your_app.conf:
+
+  Title = "Your Rails App"
+  LogFile = NONE
+
+In /etc/logwatch/scripts/services/your_app:
+
+  beaver /var/www/your_app/script/beaver.rb --yesterday /var/www/your_app/log/production.log{,.1}
+
 == Your Rails app should return appropriate HTTP statuses
 Rails does a lot of great things for us, but one thing largely up to us are
 HTTP status codes. For example, your failed logins are probably returning
-200 when they should arguably be returning 400 or 401.
+200 when they should arguably be returning 400 or 401. It's easy to do, and very useful to Beaver.
 
   render :action => :login, :status => 401
 
-See, it's easy. And very useful to Beaver.
-
-== TODO
-* Add support for Apache/Nginx/Rack::CommonLogger
-
 == License
 Copyright 2011 Jordan Hollinger
 

data/bin/beaver

@@ -4,10 +4,6 @@ require 'optparse'
 require 'rubygems'
 require 'beaver'
 
-#
-# This application was installed by RubyGems as part of the 'beaver' gem.
-#
-
 # Parse a string from the command-line into a Date object
 def parse_date(date)
   case date
@@ -19,21 +15,48 @@ end
 
 options = {}
 o = OptionParser.new do |opts|
-  opts.banner = 'Usage: beaver dams.rb [options] /path/to/rails/production.log [/another/log [/yet/another/log]]'
+  opts.banner = 'Usage: beaver [options] [dsl.rb...] /rails/production.log...'
+  opts.on('--path PATH', 'URL path, string or regex') { |path| options[:path] = Regexp.new(path, Regexp::IGNORECASE) }
+  opts.on('--method METHOD', 'HTTP request method(s), e.g. get or post,put') { |methods| options[:method] = methods.split(',').map { |m| m.downcase.to_sym } }
+  opts.on('--status STATUS', 'HTTP status(es), e.g. 200 or 500..503') { |s| options[:status] = s =~ /(\.+)/ ? Range.new(*s.split($1).map(&:to_i)) : s.to_i }
+  opts.on('--ip IP', 'IP address, string or regex') { |ip| options[:ip] = Regexp.new(ip) }
+  opts.on('--params PARAMS', 'Request parameters string (a Ruby Hash), string or regex') { |params| options[:params_str] = Regexp.new(params, Regexp::IGNORECASE) }
+  opts.on('--format FORMAT', 'Response format(s), e.g. html or json,xml') { |f| options[:format] = f.split(',').map(&:to_sym) }
+  opts.on('--longer-than MS', 'Minimum response time in ms') { |ms| options[:longer_than] = ms.to_i }
+  opts.on('--shorter-than MS', 'Maximum response time in ms') { |ms| options[:shorter_than] = ms.to_i }
   opts.on('--on DATE', 'Only include log entries from the given date (yyyy-mm-dd or -n days)') { |d| options[:on] = parse_date(d) }
   opts.on('--after DATE', 'Only include log entries from after the given date (yyyy-mm-dd or -n days)') { |d| options[:after] = parse_date(d) }
   opts.on('--before DATE', 'Only include log entries from before the given date (yyyy-mm-dd or -n days)') { |d| options[:before] = parse_date(d) }
   opts.on('--today', 'Alias to --on=-0') { options[:on] = Date.today }
   opts.on('--yesterday', 'Alias to --on=-1') { options[:on] = Date.today-1 }
+  opts.on('--regex REGEX', 'A regex string to be matched against the entire request') { |r| options[:match] = Regexp.new(r, Regexp::IGNORECASE) }
+  opts.on('--print FORMAT', 'Formatted request string, e.g. "%{ip} went to %{path} passing %{params[:email]}"') { |hit| options[:hit] = hit }
+  opts.on('-v', '--version', 'Show version') { puts "beaver #{Beaver::VERSION}"; exit }
 end
 o.parse!
 
-if ARGV.size < 2
+# Separate the beaver files and log files
+beaver_files, log_files = ARGV.uniq.partition { |a| a =~ /\.rb$/ }
+
+# There have to be log files
+if log_files.empty?
   puts o.banner
   exit 1
 end
 
-beaver_file = File.open(ARGV.shift, &:read)
-Beaver.parse *[ARGV, options].flatten do
-  eval beaver_file
+# Run logs through the DSL file(s)
+if beaver_files.any?
+  STDERR.puts "WARNING --hit is ignored when running Beaver DSL files" if options[:hit]
+
+  beaver_files.map! { |b| File.open(b, &:read) }
+  Beaver.parse *[log_files, options].flatten do
+    beaver_files.each { |b| eval b }
+  end
+# Run logs through the CLI options
+else
+  Beaver.parse *log_files do
+    hit :cli, options do
+      puts options[:hit] ? eval('"'+options[:hit].gsub(/%\{/, '#{')+'"') : to_s << $/
+    end
+  end
 end

data/lib/beaver/beaver.rb

@@ -1,6 +1,9 @@
 # Not specifically a performance analyzer (like https://github.com/wvanbergen/request-log-analyzer/wiki)
 # Rather, a DSL for finding out how people are using your Rails app (which could include performance).
 module Beaver
+  MAJOR_VERSION, MINOR_VERSION, TINY_VERSION, PRE_VERSION = 1, 0, 0, nil
+  VERSION = [MAJOR_VERSION, MINOR_VERSION, TINY_VERSION, PRE_VERSION].compact.join '.'
+
   # Alias to creating a new Beaver, parsing the files, and filtering them
   def self.parse(*args, &blk)
     raise ArgumentError, 'You must pass a block to Beaver#parse' unless block_given?
@@ -37,14 +40,14 @@ module Beaver
     # Creates a new Dam and appends it to this Beaver. name should be a unique symbol.
     # See Beaver::Dam for available options.
     def hit(dam_name, matchers={}, &callback)
-      raise ArgumentError, "A dam named #{dam_name} already exists" if @dams.has_key? dam_name
+      STDERR.puts "WARNING Overwriting existing hit '#{dam_name}'" if @dams.has_key? dam_name
       matchers = @global_matchers.merge matchers
       @dams[dam_name] = Dam.new(dam_name, matchers, &callback)
     end
 
     # Define a sumarry for a Dam
     def dam(name, &callback)
-      raise ArgumentError, "Unable to find a Dam named #{name}" unless @dams.has_key? name
+      STDERR.puts "WARNING Overwriting existing dam '#{name}'" if @sums.has_key? name
       @sums[name] = callback
     end
 
@@ -63,7 +66,11 @@ module Beaver
         end
       end
       @sums.each do |dam_name, callback|
-        @dams[dam_name].instance_eval(&callback) if @dams[dam_name].hits.any?
+        if @dams.has_key? dam_name
+          @dams[dam_name].instance_eval(&callback) if @dams[dam_name].hits.any?
+        else
+          STDERR.puts "WARNING You have defined a dam for '#{dam_name}', but there is no hit defined for '#{dam_name}'"
+        end
       end
     end
 

data/lib/beaver/dam.rb

@@ -6,11 +6,13 @@ module Beaver
   #
   #  :path => String for exact match, or Regex
   #
-  #  :method => A Symbol of :get, :post, :update or :delete, or any array of any (reads the magic _method field if present)
+  #  :method => A Symbol of :get, :post, :put or :delete, or any array of any (reads the magic _method field if present)
   #
   #  :status => A Fixnum like 404 or a Range like (500..503)
   # 
   #  :ip => String for exact match, or Regex
+  #
+  #  :format => A symbol or array of symbols of response formats like :html, :json
   # 
   #  :longer_than => Fixnum n. Matches any request which took longer than n milliseconds to complete.
   # 
@@ -65,6 +67,8 @@ module Beaver
       return false unless @match_status.nil? or @match_status === request.status
       return false unless @match_ip_s.nil? or @match_ip_s == request.ip
       return false unless @match_ip_r.nil? or @match_ip_r =~ request.ip
+      return false unless @match_format_s.nil? or @match_format_s == request.format
+      return false unless @match_format_a.nil? or @match_format_a.include? request.format
       return false unless @match_before.nil? or @match_before > request.time
       return false unless @match_after.nil? or @match_after < request.time
       return false unless @match_on.nil? or (@match_on.year == request.time.year and @match_on.month == request.time.month and @match_on.day == request.time.day)
@@ -108,8 +112,8 @@ module Beaver
       end if matchers[:path]
 
       case matchers[:method].class.name
-        when Symbol.name then @match_method_s = matchers[:method]
-        when Array.name then @match_method_a = matchers[:method]
+        when Symbol.name then @match_method_s = matchers[:method].to_s.downcase.to_sym
+        when Array.name then @match_method_a = matchers[:method].map { |m| m.to_s.downcase.to_sym }
         else raise ArgumentError, "Method must be a Symbol or an Array (it's a #{matchers[:method].class.name})"
       end if matchers[:method]
 
@@ -119,11 +123,17 @@ module Beaver
       end if matchers[:status]
 
       case matchers[:ip].class.name
-        when String.name then @match_status_s = matchers[:ip]
-        when Regexp.name then @match_status_r = matchers[:ip]
+        when String.name then @match_ip_s = matchers[:ip]
+        when Regexp.name then @match_ip_r = matchers[:ip]
         else raise ArgumentError, "IP must be a String or a Regexp (it's a #{matchers[:ip].class.name})"
       end if matchers[:ip]
 
+      case matchers[:format].class.name
+        when Symbol.name then @match_format_s = matchers[:format].to_s.downcase.to_sym
+        when Array.name then @match_format_a = matchers[:format].map { |f| f.to_s.downcase.to_sym }
+        else raise ArgumentError, "Format must be a Symbol or an Array (it's a #{matchers[:format].class.name})"
+      end if matchers[:format]
+
       case matchers[:longer_than].class.name
         when Fixnum.name then @match_longer = matchers[:longer_than]
         else raise ArgumentError, "longer_than must be a Fixnum (it's a #{matchers[:longer_than].class.name})"

data/lib/beaver/parsers/rails.rb

@@ -11,6 +11,7 @@ module Beaver
       REGEX_PATH = /^Started \w{3,4} "([^"]+)"/
       REGEX_PARAMS_STR = /^  Parameters: (\{.+\})$/
       REGEX_IP = /" for (\d+[\d.]+) at /
+      REGEX_FORMAT = /Processing by .+ as (\w+)$/
       REGEX_MS = / in (\d+)ms/
       # Depending on the version of Rails, the time format may be wildly different
       REGEX_TIME = / at ([a-z0-9:\+\- ]+)$/i
@@ -40,7 +41,7 @@ module Beaver
       def parse_method
         m = REGEX_METHOD_OVERRIDE.match(@lines)
         m = REGEX_METHOD.match(@lines) if m.nil?
-        m ? m.captures.first.downcase.to_sym : :method
+        m ? m.captures.first.downcase.to_sym : :unknown
       end
 
       # Parses and returns the response status
@@ -67,6 +68,12 @@ module Beaver
         m ? m.captures.first : BLANK_STR
       end
 
+      # Parses and returns the respones format
+      def parse_format
+        m = REGEX_FORMAT.match(@lines)
+        m ? m.captures.first.to_s.downcase.to_sym : :unknown
+      end
+
       # Parses and returns the number of milliseconds it took for the request to complete
       def parse_ms
         m = REGEX_MS.match(@lines)

data/lib/beaver/request.rb

@@ -36,7 +36,7 @@ module Beaver
 
     # Append a log line
     def <<(line)
-      @lines << line << $/
+      @lines << line
     end
 
     # Returns the request path
@@ -69,6 +69,11 @@ module Beaver
       @ip ||= parse_ip
     end
 
+    # Returns the responses format (html, json, etc)
+    def format
+      @format ||= parse_format
+    end
+
     # Returns the number of milliseconds it took for the request to complete
     def ms
       @ms ||= parse_ms
@@ -131,6 +136,11 @@ module Beaver
       BLANK_STR
     end
 
+    # Parses and returns the respones format
+    def parse_format
+      :format
+    end
+
     # Parses and returns the number of milliseconds it took for the request to complete
     def parse_ms
       0

metadata

@@ -3,10 +3,10 @@ name: beaver
 version: !ruby/object:Gem::Version 
   prerelease: false
   segments: 
+  - 1
   - 0
   - 0
-  - 1
-  version: 0.0.1
+  version: 1.0.0
 platform: ruby
 authors: 
 - Jordan Hollinger
@@ -14,11 +14,11 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2011-09-24 00:00:00 -04:00
+date: 2011-12-03 00:00:00 -05:00
 default_executable: 
 dependencies: []
 
-description: A simple DSL for helping you discover what people are doing with your Rails app
+description: A simple DSL and command-line tool for discovering what people are up to in your Rails app
 email: jordan@jordanhollinger.com
 executables: 
 - beaver
@@ -67,6 +67,6 @@ rubyforge_project:
 rubygems_version: 1.3.7
 signing_key: 
 specification_version: 3
-summary: Rails production log parser
+summary: Rails log parser
 test_files: []