yarder 0.0.2 → 0.1.0

data/README.md

@@ -7,16 +7,16 @@
 
 JSON Based Replacement logging system for Ruby on Rails.
 
-This is an experimental gem to see how easy / difficult it is to completely replace the default Ruby 
+This is an experimental gem to see how easy / difficult it is to completely replace the default Ruby
 on Rails logging system with one based on outputting JSON messages.
 
-This gem will create JSON based log entries designed for consumption by Logstash (although being 
-JSON they can be read by other software). The JSON will contain the same information as can be found 
+This gem will create JSON based log entries designed for consumption by Logstash (although being
+JSON they can be read by other software). The JSON will contain the same information as can be found
 in the default rails logging output.
 
 ## Current Status
 
-This gem is not production ready however it is probably ready for people interested in seeing the 
+This gem is not production ready however it is probably ready for people interested in seeing the
 results. All logging in a Rails3 app should be JSON formatted, including ad-hoc logging.
 
 Yarder has only been tested against Rails 3.2.8 on Ruby 1.9.3 and JRuby running in 1.9 mode. Test
@@ -26,9 +26,12 @@ unique to this gem still need to be created.
 There may be issues regarding outputting UTF-8 characters in logs on JRuby 1.6 in --1.9 mode. JRuby
 1.7 is recommended (These same issues exist in the man ruby loggers so use that as a guide).
 
-Any help, feedback or pull-requests would be much appreciated, especially related to refactoring and 
+Any help, feedback or pull-requests would be much appreciated, especially related to refactoring and
 test improvement
 
+Version 0.1.0 of this gem is designed for logstash 1.2 and Kibana version 3. If you are using an older
+version then it is probably best to stick with 0.0.2
+
 ## Installation
 
 Add this line to your Rails application's Gemfile:
@@ -57,16 +60,19 @@ module MyApp
 
     # Set a logger compatible with the standard ruby logger to be used by Yarder
     config.logger = Yarder::Logger.new(Rails.root.join('log',"#{Rails.env}.log").to_s)
-
+    # Logs AR SQL queries in debug level
+    config.logger.level = Logger::DEBUG
+    # Root name of nested logs, formerly known as :fields
+    config.logger.log_namespace = :rails
+    # Logstash logtype
+    config.logger.log_type = :rails_json_log
   end
 end
 ```
 
 ## Logstash Configuration
 
-Yarder currently creates log entries with a hard-coded logtype of "rails_json_log" (This may change 
-in future and may become configurable) therefore your Logstash configuration file should be as 
-follows:
+Yarder currently creates log entries with default logtype of "rails_json_log" therefore your Logstash configuration file should be as follows:
 
 ```
 input {
@@ -78,7 +84,14 @@ input {
 }
 ```
 
-You will need to edit the path to point to your application's log file. Because Yarder creates json 
+### Beaver
+
+```
+[/var/www/rails/application-1/log/production.log]
+format: rawjson
+```
+
+You will need to edit the path to point to your application's log file. Because Yarder creates json
 serialized Logstash::Event entries there is no need to setup any filters
 
 ### Known issues
@@ -88,7 +101,7 @@ for nested JSON but logstash web does not.
 
 ## Developers
 
-Thoughts, suggestions, opinions and contributions are welcome. 
+Thoughts, suggestions, opinions and contributions are welcome.
 
 When contributing please make sure to run your tests with warnings enabled and make sure that
 yarder creates no warnings. (Warnings from other libraries like capybara etc. are ok)
@@ -96,5 +109,3 @@ yarder creates no warnings. (Warnings from other libraries like capybara etc. ar
 ```
 RUBYOPT=-w rake
 ```
-
-

data/lib/yarder/action_controller/log_subscriber.rb

@@ -8,58 +8,50 @@ module Yarder
       def start_processing(event)
         payload = event.payload
 
-        entry.fields['controller'] = payload[:controller]
-        entry.fields['action'] = payload[:action]
-
-        format  = payload[:format]
-        entry.fields['format']  = format.to_s.downcase if format.is_a?(Symbol)
-
+        entry['name'] = payload[:controller]
+        entry['action'] = payload[:action]
+        entry['path'] = payload[:path]
+        entry['format']  = payload[:format].to_s.downcase
       end
 
       def process_action(event)
-
         payload   = event.payload
-        #TODO Think about additions. Comment out for the moment to shut up warnings
-        #additions = ::ActionController::Base.log_process_action(payload)
 
         params = payload[:params].except(*INTERNAL_PARAMS)
-        entry.fields['parameters'] = params unless params.empty?
-
-        entry.fields['controller_duration'] = event.duration
-
-        #TODO  What on earth are additions and how should we handle them?
-        # message << " (#{additions.join(" | ")})" unless additions.blank?
+        entry['parameters'] = params unless params.empty?
 
+        root['duration']['controller'] = event.duration
       end
 
       def halted_callback(event)
-        entry.fields['halted_callback'] = event.payload[:filter]
+        entry['halted_callback'] = event.payload[:filter]
       end
 
       def send_file(event)
-        entry.fields['send_file'] = event.payload[:path]
-        entry.fields['send_file_duration'] = event.duration
+        entry['send_file'] = event.payload[:path]
+        root['duration']['send_file'] = event.duration
       end
 
       def redirect_to(event)
-        entry.fields['redirect_to'] = event.payload[:location]
+        entry['redirect_to'] = event.payload[:location]
       end
 
       def send_data(event)
-        entry.fields['send_data'] = event.payload[:filename]
-        entry.fields['send_data_duration'] = event.duration
+        entry['send_data'] = event.payload[:filename]
+        root['duration']['send_data'] = event.duration
       end
 
       %w(write_fragment read_fragment exist_fragment?
          expire_fragment expire_page write_page).each do |method|
         class_eval <<-METHOD, __FILE__, __LINE__ + 1
           def #{method}(event)
-            entry.fields['cache'] ||= []
             cache_event = {}
             cache_event['key_or_path'] = event.payload[:key] || event.payload[:path]
             cache_event['type'] = #{method.to_s.humanize.inspect}
             cache_event['duration'] = event.duration
-            entry.fields['cache'] << cache_event
+            cache << cache_event
+            root['duration']['cache'] ||= 0
+            root['duration']['cache'] += event.duration.to_f
           end
         METHOD
       end
@@ -67,7 +59,15 @@ module Yarder
     private
 
       def entry
-        Yarder.log_entries[Thread.current]
+        @entry ||= (root['controller'] ||= {})
+      end
+
+      def cache
+        @cache ||= (root['cache'] ||= [])
+      end
+
+      def root
+        @root ||= Yarder.log_entries[Thread.current].fields.tap { |o| o['duration'] ||= {} }
       end
 
     end

data/lib/yarder/action_view/log_subscriber.rb

@@ -4,14 +4,17 @@ module Yarder
     class LogSubscriber < ::ActiveSupport::LogSubscriber
 
       def render_template(event)
-        entry.fields['rendering'] ||= []
         render_entry = {}
         render_entry['identifier'] = from_rails_root(event.payload[:identifier])
         render_entry['layout'] = from_rails_root(event.payload[:layout]) if event.payload[:layout]
         render_entry['duration'] = event.duration
 
+        entry.fields['rendering'] ||= []
         entry.fields['rendering'] << render_entry
 
+        entry.fields['duration'] ||= {}
+        entry.fields['duration']['rendering'] ||= 0
+        entry.fields['duration']['rendering'] += event.duration.to_f
       end
       alias :render_partial :render_template
       alias :render_collection :render_template

data/lib/yarder/active_record/log_subscriber.rb

@@ -66,6 +66,10 @@ module Yarder
         entry = log_entry
         entry.fields['sql'] ||= []
         entry.fields['sql'] << sql_entry
+
+        entry.fields['duration'] ||= {}
+        entry.fields['duration']['sql'] ||= 0
+        entry.fields['duration']['sql'] += sql_entry['duration'].to_f
         entry.write(false)
       end
 
@@ -74,12 +78,12 @@ module Yarder
       end
 
       def log_entry
-        Yarder.log_entries[Thread.current] ||
-          Yarder::Event.new(Rails.logger, false).tap do |entry|
-          entry.fields['uuid'] = SecureRandom.uuid
-          #TODO Should really move this into the base logger
-          entry.source = Socket.gethostname
-          entry.type = "rails_json_log"
+        Yarder.log_entries[Thread.current] || Yarder::Event.create(Rails.logger, tags)
+      end
+
+      def tags
+        if log_tags = Rails.configuration.log_tags
+          log_tags.select{|tag| tag.is_a? String}
         end
       end
 

data/lib/yarder/active_resource/log_subscriber.rb

@@ -3,12 +3,7 @@ module Yarder
     class LogSubscriber < ::ActiveSupport::LogSubscriber
 
       def request(event)
-
-        #TODO Think of a better name for this!
-        entry.fields['active_resource'] ||= []
-
         request_entry = {}
-
         request_entry['method'] = event.payload[:method].to_s.upcase
         request_entry['uri'] = event.payload[:request_uri]
 
@@ -19,7 +14,12 @@ module Yarder
         request_entry['length'] = result.body.to_s.length
         request_entry['duration'] = event.duration
 
-        entry.fields['active_resource'] << request_entry
+        entry.fields['rest'] ||= []
+        entry.fields['rest'] << request_entry
+
+        entry.fields['duration'] ||= {}
+        entry.fields['duration']['rest'] ||= 0
+        entry.fields['duration']['rest'] += event.duration.to_f
       end
 
     private
@@ -31,4 +31,3 @@ module Yarder
     end
   end
 end
-

data/lib/yarder/event.rb

@@ -3,14 +3,18 @@ module Yarder
   # Basically a wrapper for a LogStash event that keeps track of if it was created from a rack
   # middle-ware or not. This is important when it comes to deciding when to write the log
   class Event
-
     extend Forwardable
-    def_delegators :@logstash_event, :fields, :message=, :source=, :type=, :tags, :to_json
+    def_delegators :@logstash_event, :[]=, :[], :to_json
 
     def initialize(logger, rack = false)
       @logger = logger
       @rack = rack
       @logstash_event = LogStash::Event.new
+
+      self['type'] = logger.log_type
+      self['tags'] ||= []
+      self.fields['duration'] = {}
+      self.fields['env'] = logger.env
     end
 
     def write(rack = false)
@@ -21,24 +25,14 @@ module Yarder
       end
     end
 
-    def add_tags_to_logger(request, tags)
-      tag_hash = []
-      if tags
-        tags.each do |tag|
-          case tag
-          when Symbol
-            tag_hash << {tag.to_s => request.send(tag) }
-          when Proc
-            tag_hash << tag.call(request)
-          else
-            tag_hash << tag
-          end
-        end
-      end
-
-      @logger.push_request_tags(tag_hash)
+    def self.create(logger, tags, rack = false)
+      logger.push_request_tags(tags) if tags
+      new(logger, rack)
     end
 
+    def fields
+      @fields ||= (@logstash_event[@logger.log_namespace.to_s] ||= {})
+    end
   end
 
-end
+end

data/lib/yarder/logger.rb

@@ -1,4 +1,5 @@
 require 'logger'
+require 'socket'
 
 module Yarder
 
@@ -39,11 +40,26 @@ module Yarder
       end
     end
 
+    attr_accessor :log_type, :log_namespace
+
     def initialize(*args)
       super
+      self.log_type = :rails_json_log
+      self.log_namespace = :rails
       @formatter = SimpleFormatter.new
     end
 
+    def env
+      @env ||= {
+        :ruby => "#{RUBY_VERSION}-p#{RUBY_PATCHLEVEL}",
+        :env => Rails.env,
+        :pwd => Dir.pwd,
+        :program => $0,
+        :user => ENV['USER'],
+        :host => ::Socket.gethostname
+      }
+    end
+
     # Simple formatter which only displays the message.
     class SimpleFormatter < ::Logger::Formatter
       # This method is invoked when a log event occurs
@@ -52,4 +68,4 @@ module Yarder
       end
     end
   end
-end
+end

data/lib/yarder/rack/logger.rb

@@ -13,40 +13,43 @@ module Yarder
         t1 = Time.now
         request = ActionDispatch::Request.new(env)
 
-        event = Yarder::Event.new(Rails.logger, true)
-        event.message = "#{request.request_method} #{request.filtered_path} for #{request.ip}"
-        event.fields['client_ip'] = request.ip
-        event.fields['method'] = request.request_method
-        event.fields['path'] = request.filtered_path
-        #TODO Should really move this into the base logger
-        event.source = "http://#{Socket.gethostname}#{request.filtered_path}"
-        event.type = "rails_json_log"
+        event = Yarder::Event.create Rails.logger, tags(request), true
+        event['message'] = "#{request.request_method} #{request.filtered_path} for #{request.ip}"
 
-        event.add_tags_to_logger(request, @tags) if @tags
+        entry = (event.fields['rack'] ||= {})
+        entry['client_ip'] = request.ip
+        entry['method'] = request.request_method
+        entry['path'] = request.filtered_path
+        entry['url'] = request.url
 
         Yarder.log_entries[Thread.current] = event
 
         status, headers, response = @app.call(env)
         [status, headers, response]
-
       ensure
         if event
-          event.fields['total_duration'] = Time.now - t1
-          event.fields['status'] = status
-
-          ['rendering','sql'].each do |type|
-            if event.fields[type] && !event.fields[type].empty?
-              duration = event.fields[type].inject(0) {|result, local_event| result += local_event[:duration].to_f }
-              event.fields["#{type}_duration"] = duration
-            end
-          end
-
+          entry['status'] = status
+          event.fields['duration']['total'] = (Time.now - t1)*1000
           event.write(true)
         end
 
         Yarder.log_entries[Thread.current] = nil
       end
 
+      def tags(request)
+        return unless @tags
+        @tags.reduce([]) do |arr, tag|
+          case tag
+          when Symbol
+            arr << {tag.to_s => request.send(tag) }
+          when Proc
+            arr << tag.call(request)
+          else
+            arr << tag
+          end
+        end
+      end
+
     end
 
   end

data/lib/yarder/tagged_logging.rb

@@ -24,9 +24,10 @@ module Yarder
           @entry = msg
         else
           @entry = Yarder::Event.new(Rails.logger)
-          @entry.message = msg
+          @entry['message'] = msg
         end
-        @entry.fields['severity'] = severity
+        @entry['severity'] = severity
+
         process_tags(current_tags)
         process_tags(current_request_tags)
         #TODO Should we do anything with progname? What about source?
@@ -57,7 +58,7 @@ module Yarder
               @entry.fields[k] = v
             end
           else
-            @entry.tags << tag
+            @entry['tags'] << tag
           end
         end
       end

data/lib/yarder/version.rb

@@ -1,3 +1,3 @@
 module Yarder
-  VERSION = "0.0.2"
+  VERSION = "0.1.0"
 end

data/test/action_controller/log_subscriber_test.rb

@@ -28,12 +28,13 @@ class ACLogSubscriberTest < ActionController::TestCase
   end
 
   def test_start_processing
-    get :show, {:test => 'test'}
+    get :show, {:attr => 'test'}
     wait
 
-    assert_equal "LogSubscribersController", @log_entry.fields['controller']
-    assert_equal "show", @log_entry.fields['action']
-    assert_equal "html", @log_entry.fields['format']
+    assert_equal "LogSubscribersController", @log_entry.fields['controller']['name']
+    assert_equal "show", @log_entry.fields['controller']['action']
+    assert_equal "/show?attr=test", @log_entry.fields['controller']['path']
+    assert_equal "html", @log_entry.fields['controller']['format']
   end
 
 
@@ -41,28 +42,28 @@ class ACLogSubscriberTest < ActionController::TestCase
     get :never_executed
     wait
 
-    assert_equal ":redirector" ,@log_entry.fields['halted_callback']
+    assert_equal ":redirector" ,@log_entry.fields['controller']['halted_callback']
   end
 
-  def test_process_action
+  def test_controller_duration_action
     get :show
     wait
 
-    assert_present @log_entry.fields['controller_duration']
+    assert_present @log_entry.fields['duration']['controller']
   end
 
   def test_process_action_without_parameters
     get :show
     wait
 
-    assert_blank @log_entry.fields['parameters']
+    assert_blank @log_entry.fields['controller']['parameters']
   end
 
   def test_process_action_with_parameters
     get :show, :id => '10'
     wait
 
-    assert_equal '10', @log_entry.fields['parameters']['id']
+    assert_equal '10', @log_entry.fields['controller']['parameters']['id']
   end
 
   def test_process_action_with_wrapped_parameters
@@ -70,8 +71,8 @@ class ACLogSubscriberTest < ActionController::TestCase
     post :show, :id => '10', :name => 'jose'
     wait
 
-    assert_equal '10', @log_entry.fields['parameters']['id']
-    assert_equal 'jose', @log_entry.fields['parameters']['name']
+    assert_equal '10', @log_entry.fields['controller']['parameters']['id']
+    assert_equal 'jose', @log_entry.fields['controller']['parameters']['name']
   end
 
   def test_process_action_with_filter_parameters
@@ -80,7 +81,7 @@ class ACLogSubscriberTest < ActionController::TestCase
     get :show, :lifo => 'Pratik', :amount => '420', :step => '1'
     wait
 
-    params = @log_entry.fields['parameters']
+    params = @log_entry.fields['controller']['parameters']
     assert_equal '[FILTERED]', params['amount']
     assert_equal '[FILTERED]', params['lifo']
     assert_equal '1', params['step']
@@ -90,7 +91,7 @@ class ACLogSubscriberTest < ActionController::TestCase
     get :redirector
     wait
 
-    assert_equal 'http://foo.bar/', @log_entry.fields['redirect_to']
+    assert_equal 'http://foo.bar/', @log_entry.fields['controller']['redirect_to']
   end
 
 
@@ -98,8 +99,8 @@ class ACLogSubscriberTest < ActionController::TestCase
     get :data_sender
     wait
 
-    assert_equal 'file.txt', @log_entry.fields['send_data']
-    assert_present @log_entry.fields['send_data_duration']
+    assert_equal 'file.txt', @log_entry.fields['controller']['send_data']
+    assert_present @log_entry.fields['duration']['send_data']
   end
 
 
@@ -107,8 +108,8 @@ class ACLogSubscriberTest < ActionController::TestCase
     get :file_sender
     wait
 
-    assert_match 'test/dummy/public/favicon.ico', @log_entry.fields['send_file']
-    assert_present @log_entry.fields['send_file_duration']
+    assert_match 'test/dummy/public/favicon.ico', @log_entry.fields['controller']['send_file']
+    assert_present @log_entry.fields['duration']['send_file']
   end
 
   def test_with_fragment_cache
@@ -123,6 +124,8 @@ class ACLogSubscriberTest < ActionController::TestCase
 
     assert_match('Write fragment', @log_entry.fields['cache'].last['type'])
     assert_match('views/foo', @log_entry.fields['cache'].last['key_or_path'])
+
+    assert_present @log_entry.fields['duration']['cache']
   ensure
     LogSubscribersController.config.perform_caching = true
   end
@@ -144,22 +147,20 @@ class ACLogSubscriberTest < ActionController::TestCase
     LogSubscribersController.config.perform_caching = true
   end
 
-=begin TODO Figure out why this last test fails.
   def test_with_page_cache
     @controller.config.perform_caching = true
     get :with_page_cache
     wait
 
-   assert_present @log_entry.fields['cache']
+    assert_present @log_entry.fields['cache']
 
-    assert_match('Write page', @log_entry.fields['cache'][1]['type'])
-    assert_match('index.html', @log_entry.fields['cache'][1]['key_or_path'])
+    assert_match('Write page', @log_entry.fields['cache'].first['type'])
+    assert_match('index.html', @log_entry.fields['cache'].first['key_or_path'])
   ensure
     @controller.config.perform_caching = true
   end
-=end
 
   def logs
     @logs ||= @logger.logged(:info)
   end
-end
+end