lograge 0.0.3 → 0.1.0

data/Gemfile

@@ -2,3 +2,8 @@ source "http://rubygems.org"
 
 # Specify your gem's dependencies in lograge.gemspec
 gemspec
+
+group :test do
+  gem 'actionpack'
+  gem 'logstash-event'
+end

data/README.md

@@ -7,6 +7,10 @@ default logging output. Rails' default approach to log everything is great
 during development, it's terrible when running it in production. It pretty much
 renders Rails logs useless to me.
 
+Lograge is a work in progress. I appreciate constructive feedback and criticism.
+My main goal is to improve Rails' logging and to show people that they don't
+need to stick with its defaults anymore if they don't want to.
+
 Instead of trying solving the problem of having multiple lines per request by
 switching Rails' logger for something that outputs syslog lines or adds a
 request token, Lograge replaces Rails' request logging entirely, reducing the
@@ -30,7 +34,7 @@ Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord: 0.0ms)
 you get a single line with all the important information, like this:
 
 ```
-GET /jobs/833552.json format=json action=jobs#show status=200 duration=58.33 view=40.43 db=15.26
+method=GET path=/jobs/833552.json format=json controller=jobs action=show status=200 duration=58.33 view=40.43 db=15.26
 ```
 
 The second line is easy to grasp with a single glance and still includes all the
@@ -51,10 +55,44 @@ Enable it for the relevant environments, e.g. production:
 ```
 # config/environments/production.rb
 MyApp::Application.configure do
-  config.lograge.enabled = true 
+  config.lograge.enabled = true
+end
+```
+
+You can also add a hook for own custom data
+
+```
+# config/environments/staging.rb
+MyApp::Application.configure do
+  config.lograge.enabled = true
+
+  # custom_options can be a lambda or hash
+  # if it's a lambda then it must return a hash
+  config.lograge.custom_options = lambda do |event|
+    # capture some specific timing values you are interested in
+    {:name => "value", :timing => some_float.round(2)}
+  end
+end
+```
+
+Lograge supports multiple output formats. The most common is the default
+lograge format described above. Alternatively, you can also generate JSON
+logs in the json_event format used by [Logstash](http://logstash.net/).
+
+```
+# config/environments/production.rb
+MyApp::Application.configure do
+  config.lograge.log_format = :logstash
 end
 ```
 
+*Note:* When using the logstash output, you need to add the additional gem
+`logstash-event`. You can simply add it to your Gemfile like this
+
+```ruby
+gem "logstash-event"
+```
+
 Done.
 
 **Internals**
@@ -79,16 +117,24 @@ While the LogSubscribers encapsulate most logging pretty nicely, there are still
 two lines that show up no matter what. The first line that's output for every
 Rails request, you know, this one:
 
+```
+Started GET "/" for 127.0.0.1 at 2012-03-12 17:10:10 +0100
+```
+
 And the verbose output coming from rack-cache:
 
+```
+cache: [GET /] miss
+```
+
 Both are independent of the LogSubscribers, and both need to be shut up using
 different means.
 
-For the first one, the starting line of every Rails request log, Lograge removes
-the `Rails::Rack::Logger` middleware from the stack. This may look like a drastic
-means, but all the middleware does is log that useless line, log exceptions, and
-create a request transaction id (Rails 3.2). A future version may replace with
-its own middleware, that simply removes the log line.
+For the first one, the starting line of every Rails request log, Lograge
+replaces code in `Rails::Rack::Logger` to remove that particular log line. It's
+not great, but it's just another unnecessary output and would still clutter the
+log files. Maybe a future version of Rails will make this log line an event as
+well.
 
 To remove rack-cache's output (which is only enabled if caching in Rails is
 enabled), Lograge disables verbosity for rack-cache, which is unfortunately
@@ -98,6 +144,9 @@ There, a single line per request. Beautiful.
 
 **What it doesn't do**
 
+Lograge is opinionated, very opinionated. If the stuff below doesn't suit your
+needs, it may not be for you.
+
 Lograge removes ActionView logging, which also includes rendering times for
 partials. If you're into those, Lograge is probably not for you. In my honest
 opinion, those rendering times don't belong in the log file, they should be
@@ -106,6 +155,18 @@ service that allows graphing rendering percentiles. I assume this for everything
 that represents a moving target. That kind of data is better off being
 visualized in graphs than dumped (and ignored) in a log file.
 
+Lograge doesn't yet log the request parameters. This is something I'm actively
+contemplating, mainly because I want to find a good way to include them, a way
+that fits in with the general spirit of the log output generated by Lograge.
+
+**Changes**
+
+* Add support for Logstash events (Holger Just, http://github.com/meineerde)
+* Fix for Rails 3.2.9
+* Use keys everywhere (Curt Michols, http://github.com/asenchi)
+* Add `custom_options` to allow adding custom key-value pairs at runtime (Adam
+  Cooper, https://github.com/adamcooper)
+
 **License**
 
 MIT. Code extracted from [Travis CI](http://travis-ci.org).

data/lib/lograge/log_subscriber.rb

@@ -5,11 +5,42 @@ module Lograge
   class RequestLogSubscriber < ActiveSupport::LogSubscriber
     def process_action(event)
       payload = event.payload
-      message = "#{payload[:method]} #{payload[:path]} format=#{payload[:format]} action=#{payload[:params]['controller']}##{payload[:params]['action']}"
-      message << extract_status(payload)
-      message << runtimes(event)
-      message << location(event)
-      logger.info(message)
+
+      data      = extract_request(payload)
+      data.merge! extract_status(payload)
+      data.merge! runtimes(event)
+      data.merge! location(event)
+      data.merge! custom_options(event)
+
+      logger.info send(:"process_action_#{Lograge.log_format}", data)
+    end
+
+    LOGRAGE_FIELDS = [
+      :method, :path, :format, :controller, :action, :status, :error,
+      :duration, :view, :db, :location
+    ]
+    def process_action_lograge(data)
+      fields  = LOGRAGE_FIELDS
+      fields += (data.keys - LOGRAGE_FIELDS)
+
+      event = fields.inject([]) do |message, key|
+        next message unless data.has_key?(key)
+        # Exactly preserve the previous output
+        # Parsing this can be ambigious if the error messages contains
+        # a single quote
+        data[key] = "'#{data[key]}'" if key == :error
+        # Ensure that we always have exactly two decimals
+        data[key] = "%.2f" % data[key] if data[key].is_a? Numeric
+
+        message << "#{key}=#{data[key]}"
+        message
+      end
+      event.join(" ")
+    end
+
+    def process_action_logstash(data)
+      event = LogStash::Event.new("@fields" => data)
+      event.to_json
     end
 
     def redirect_to(event)
@@ -18,32 +49,57 @@ module Lograge
 
     private
 
+    def extract_request(payload)
+      {
+        :method => payload[:method],
+        :path => payload[:path],
+        :format => extract_format(payload),
+        :controller => payload[:params]['controller'],
+        :action => payload[:params]['action']
+      }
+    end
+
+    def extract_format(payload)
+      if ::ActionPack::VERSION::MAJOR == 3 && ::ActionPack::VERSION::MINOR == 0
+        payload[:formats].first
+      else
+        payload[:format]
+      end
+    end
+
     def extract_status(payload)
       if payload[:status]
-        " status=#{payload[:status]}"
+        { :status => payload[:status].to_i }
       elsif payload[:exception]
         exception, message = payload[:exception]
-        " status=500 error='#{exception}:#{message}'"
+        { :status => 500, :error => "#{exception}:#{message}" }
+      else
+        { :status => 0 }
       end
     end
 
+    def custom_options(event)
+      Lograge.custom_options(event) || {}
+    end
+
     def runtimes(event)
-      message = ""
-      {:duration => event.duration,
-       :view => event.payload[:view_runtime],
-       :db => event.payload[:db_runtime]}.each do |name, runtime|
-        message << " #{name}=%.2f" % runtime if runtime
+      {
+        :duration => event.duration,
+        :view => event.payload[:view_runtime],
+        :db => event.payload[:db_runtime]
+      }.inject({}) do |runtimes, (name, runtime)|
+        runtimes[name] = runtime.to_f.round(2) if runtime
+        runtimes
       end
-      message
     end
 
     def location(event)
       if location = Thread.current[:lograge_location]
         Thread.current[:lograge_location] = nil
-        "location=#{location}"
+        { :location => location }
       else
-        ""
+        {}
       end
-    end  
+    end
   end
 end

data/lib/lograge/rails_ext/rack/logger.rb

@@ -9,7 +9,8 @@ module Rails
     # Started GET / for 192.168.2.1...
     class Logger
       # Overwrites Rails 3.2 code that logs new requests
-      def call_app(env)
+      def call_app(*args)
+        env = args.last
         @app.call(env)
       ensure
         ActiveSupport::LogSubscriber.flush_all!

data/lib/lograge/version.rb

@@ -1,3 +1,3 @@
 module Lograge
-  VERSION = "0.0.3"
+  VERSION = "0.1.0"
 end

data/lib/lograge.rb

@@ -5,32 +5,71 @@ require 'active_support/core_ext/string/inflections'
 require 'active_support/ordered_options'
 
 module Lograge
-  mattr_accessor :logger  
+  mattr_accessor :logger
 
-  def self.remove_existing_log_subscriptions
-    %w(redirect_to process_action start_processing send_data write_fragment exist_fragment? send_file).each do |event|
-      unsubscribe_from_event(:action_controller, event)
+  # Custom options that will be appended to log line
+  #
+  # Currently supported formats are:
+  #  - Hash
+  #  - Any object that responds to call and returns a hash
+  #
+  mattr_writer :custom_options
+  self.custom_options = nil
+
+  def self.custom_options(event)
+    if @@custom_options.respond_to?(:call)
+      @@custom_options.call(event)
+    else
+      @@custom_options
     end
+  end
+
+  # The emitted log format
+  #
+  # Currently supported formats are>
+  #  - :lograge - The custom tense lograge format
+  #  - :logstash - JSON formatted as a Logstash Event.
+  mattr_accessor :log_format
+  self.log_format = :lograge
 
-    %w{render_template render_partial render_collection}.each do |event|
-      unsubscribe_from_event(:action_view, event) 
+  def self.remove_existing_log_subscriptions
+    ActiveSupport::LogSubscriber.log_subscribers.each do |subscriber|
+      case subscriber
+      when ActionView::LogSubscriber
+        unsubscribe(:action_view, subscriber)
+      when ActionController::LogSubscriber
+        unsubscribe(:action_controller, subscriber)
+      end
     end
   end
 
-  def self.unsubscribe_from_event(component, event)
-    delegate_type = component.to_s.classify
-    ActiveSupport::Notifications.notifier.listeners_for("#{event}.#{component}").each do |listener|
-      if listener.inspect =~ /delegate[^a-z]+#{delegate_type}/
-        ActiveSupport::Notifications.unsubscribe listener
+  def self.unsubscribe(component, subscriber)
+    events = subscriber.public_methods(false).reject{ |method| method.to_s == 'call' }
+    events.each do |event|
+      ActiveSupport::Notifications.notifier.listeners_for("#{event}.#{component}").each do |listener|
+        if listener.instance_variable_get('@delegate') == subscriber
+          ActiveSupport::Notifications.unsubscribe listener
+        end
       end
     end
   end
 
   def self.setup(app)
-    app.config.action_dispatch.rack_cache[:verbose] = false
+    app.config.action_dispatch.rack_cache[:verbose] = false if app.config.action_dispatch.rack_cache
     require 'lograge/rails_ext/rack/logger'
     Lograge.remove_existing_log_subscriptions
     Lograge::RequestLogSubscriber.attach_to :action_controller
+    Lograge.custom_options = app.config.lograge.custom_options
+    Lograge.log_format = app.config.lograge.log_format || :lograge
+    case Lograge.log_format.to_s
+    when "logstash"
+      begin
+        require "logstash-event"
+      rescue LoadError
+        puts "You need to install the logstash-event gem to use the logstash output."
+        raise
+      end
+    end
   end
 end
 

data/lograge.gemspec

@@ -7,7 +7,7 @@ Gem::Specification.new do |s|
   s.version     = Lograge::VERSION
   s.authors     = ["Mathias Meyer"]
   s.email       = ["meyer@paperplanes.de"]
-  s.homepage    = ""
+  s.homepage    = "https://github.com/roidrage/lograge"
   s.summary     = %q{Tame Rails' multi-line logging into a single line per request}
   s.description = %q{Tame Rails' multi-line logging into a single line per request}
 

data/spec/lograge_logsubscriber_spec.rb

@@ -1,4 +1,5 @@
 require 'spec_helper'
+require 'lograge'
 require 'lograge/log_subscriber'
 require 'active_support/notifications'
 require 'active_support/core_ext/string'
@@ -14,7 +15,7 @@ describe Lograge::RequestLogSubscriber do
     logger
   }
   before do
-    Lograge::RequestLogSubscriber.logger = logger    
+    Lograge::RequestLogSubscriber.logger = logger
   end
 
   let(:subscriber) {Lograge::RequestLogSubscriber.new}
@@ -34,7 +35,11 @@ describe Lograge::RequestLogSubscriber do
     )
   }
 
-  describe "when processing an action" do
+  describe "when processing an action with lograge output" do
+    before do
+      Lograge::log_format = :lograge
+    end
+
     it "should include the URL in the log output" do
       subscriber.process_action(event)
       log_output.string.should include('/home')
@@ -42,9 +47,9 @@ describe Lograge::RequestLogSubscriber do
 
     it "should start the log line with the HTTP method" do
       subscriber.process_action(event)
-      log_output.string.starts_with?('GET').should == true
+      log_output.string.starts_with?('method=GET').should == true
     end
-    
+
     it "should include the status code" do
       subscriber.process_action(event)
       log_output.string.should include('status=200')
@@ -52,7 +57,7 @@ describe Lograge::RequestLogSubscriber do
 
     it "should include the controller and action" do
       subscriber.process_action(event)
-      log_output.string.should include('action=home#index')
+      log_output.string.should include('controller=home action=index')
     end
 
     it "should include the duration" do
@@ -78,6 +83,13 @@ describe Lograge::RequestLogSubscriber do
       log_output.string.should =~ /error='AbstractController::ActionNotFound:Route not found'/
     end
 
+    it "should return an unknown status when no status or exception is found" do
+      event.payload[:status] = nil
+      event.payload[:exception] = nil
+      subscriber.process_action(event)
+      log_output.string.should =~ /status=0/
+    end
+
     describe "with a redirect" do
       before do
         Thread.current[:lograge_location] = "http://www.example.com"
@@ -85,7 +97,7 @@ describe Lograge::RequestLogSubscriber do
 
       it "should add the location to the log line" do
         subscriber.process_action(event)
-        log_output.string.should =~ %r{location=http://www.example.com}
+        log_output.string.should =~ %r{ location=http://www.example.com}
       end
 
       it "should remove the thread local variable" do
@@ -93,13 +105,136 @@ describe Lograge::RequestLogSubscriber do
         Thread.current[:lograge_location].should == nil
       end
     end
-    
+
     it "should not include a location by default" do
       subscriber.process_action(event)
       log_output.string.should_not =~ /location=/
     end
   end
 
+    describe "when processing an action with logstash output" do
+    before do
+      require 'logstash-event'
+      Lograge::log_format = :logstash
+    end
+
+    it "should include the URL in the log output" do
+      subscriber.process_action(event)
+      log_output.string.should include('/home')
+    end
+
+    it "should start include the HTTP method" do
+      subscriber.process_action(event)
+      log_output.string.should include('"method":"GET"')
+    end
+
+    it "should include the status code" do
+      subscriber.process_action(event)
+      log_output.string.should include('"status":200')
+    end
+
+    it "should include the controller and action" do
+      subscriber.process_action(event)
+      log_output.string.should include('"controller":"home"')
+      log_output.string.should include('"action":"index"')
+    end
+
+    it "should include the duration" do
+      subscriber.process_action(event)
+      log_output.string.should =~ /"duration":\d+\.\d{0,2}/
+    end
+
+    it "should include the view rendering time" do
+      subscriber.process_action(event)
+      log_output.string.should =~ /"view":0.01/
+    end
+
+    it "should include the database rendering time" do
+      subscriber.process_action(event)
+      log_output.string.should =~ /"db":0.02/
+    end
+
+    it "should add a 500 status when an exception occurred" do
+      event.payload[:status] = nil
+      event.payload[:exception] = ['AbstractController::ActionNotFound', 'Route not found']
+      subscriber.process_action(event)
+      log_output.string.should =~ /"status":500/
+      log_output.string.should =~ /"error":"AbstractController::ActionNotFound:Route not found"/
+    end
+
+    it "should return an unknown status when no status or exception is found" do
+      event.payload[:status] = nil
+      event.payload[:exception] = nil
+      subscriber.process_action(event)
+      log_output.string.should =~ /"status":0/
+    end
+
+    describe "with a redirect" do
+      before do
+        Thread.current[:lograge_location] = "http://www.example.com"
+      end
+
+      it "should add the location to the log line" do
+        subscriber.process_action(event)
+        log_output.string.should =~ %r{"location":"http://www.example.com"}
+      end
+
+      it "should remove the thread local variable" do
+        subscriber.process_action(event)
+        Thread.current[:lograge_location].should == nil
+      end
+    end
+
+    it "should not include a location by default" do
+      subscriber.process_action(event)
+      log_output.string.should_not =~ /"location":/
+    end
+  end
+
+  describe "with custom_options configured for lograge output" do
+    before do
+      Lograge::log_format = :lograge
+    end
+
+    it "should combine the hash properly for the output" do
+      Lograge.custom_options = {:data => "value"}
+      subscriber.process_action(event)
+      log_output.string.should =~ / data=value/
+    end
+    it "should combine the output of a lambda properly" do
+      Lograge.custom_options = lambda {|event| {:data => "value"}}
+      subscriber.process_action(event)
+      log_output.string.should =~ / data=value/
+    end
+    it "should work if the method returns nil" do
+      Lograge.custom_options = lambda {|event| nil}
+      subscriber.process_action(event)
+      log_output.string.should be_present
+    end
+  end
+
+  describe "with custom_options configured for logstash output" do
+    before do
+      Lograge::log_format = :logstash
+    end
+
+    it "should combine the hash properly for the output" do
+      Lograge.custom_options = {:data => "value"}
+      subscriber.process_action(event)
+      log_output.string.should =~ /"data":"value"/
+    end
+    it "should combine the output of a lambda properly" do
+      Lograge.custom_options = lambda {|event| {:data => "value"}}
+      subscriber.process_action(event)
+      log_output.string.should =~ /"data":"value"/
+    end
+    it "should work if the method returns nil" do
+      Lograge.custom_options = lambda {|event| nil}
+      subscriber.process_action(event)
+      log_output.string.should be_present
+    end
+  end
+
   describe "when processing a redirect" do
     it "should store the location in a thread local variable" do
       subscriber.redirect_to(redirect)

data/spec/spec_helper.rb

@@ -4,6 +4,8 @@
 # loaded once.
 #
 # See http://rubydoc.info/gems/rspec-core/RSpec/Core/Configuration
+require 'action_pack'
+
 RSpec.configure do |config|
   config.treat_symbols_as_metadata_keys_with_true_values = true
   config.run_all_when_everything_filtered = true

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: lograge
 version: !ruby/object:Gem::Version
-  version: 0.0.3
+  version: 0.1.0
   prerelease: 
 platform: ruby
 authors:
@@ -9,11 +9,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-03-11 00:00:00.000000000 Z
+date: 2012-11-13 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rspec
-  requirement: &70153872860480 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -21,10 +21,15 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70153872860480
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: guard-rspec
-  requirement: &70153872859680 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -32,10 +37,15 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70153872859680
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: activesupport
-  requirement: &70153872858340 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -43,10 +53,15 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *70153872858340
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: actionpack
-  requirement: &70153872856860 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -54,7 +69,12 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *70153872856860
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 description: Tame Rails' multi-line logging into a single line per request
 email:
 - meyer@paperplanes.de
@@ -78,7 +98,7 @@ files:
 - spec/lograge_logsubscriber_spec.rb
 - spec/lograge_spec.rb
 - spec/spec_helper.rb
-homepage: ''
+homepage: https://github.com/roidrage/lograge
 licenses: []
 post_install_message: 
 rdoc_options: []
@@ -98,7 +118,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: lograge
-rubygems_version: 1.8.11
+rubygems_version: 1.8.23
 signing_key: 
 specification_version: 3
 summary: Tame Rails' multi-line logging into a single line per request